AWS & Databricks, Data Lead

Job summary
Collaborate with cross-functional teams to understand data requirements and translate them into scalable AWS Data solution. Integrate data from various sources into Databricks clusters. Design, implement, and maintain data processing pipelines using Databricks and Apache Spark.
Job seniority: mid-to-senior level
Responsibilities
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable AWS Data solution. • Integrate data from various sources, including data lakes, databases, and external APIs, into Databricks clusters. • Design, implement, and maintain data processing pipelines using Databricks and Apache Spark to handle large volume data migration. • Implement security measures and access controls to ensure data privacy and compliance with industry regulations. • Troubleshoot and resolve technical issues related to data processing and Databricks platform functionality. • Automate data workflows, job scheduling, and routine maintenance tasks. • Implement Data Governance using AWS native and Databricks services.
Requirements
• Bachelor's degree in Computer Science, Data Science, or a related field. • Proven experience as a AWS Databricks Engineer with a minimum of 8 years’ experience with building ETL pipeline processes (pySpark), data lake (AWS S3, Delta lake) and workflow orchestration (Apache Airflow). • Proficiency in programming languages such as pySpark. • Performed change data capture, complex data transformation, file format conversions to Delta lake tables and implemented slowly changing dimensions using Databricks pipeline. • Hands on experience in orchestrating Databricks workflow using Amazon Managed Workflows for Apache Airflow. • Hands on experience in Delta lake table integrations with AWS Glue catalog and AWS Athena. • AWS Athena JDBC java programing is plus. • Strong problem-solving skills and the ability to troubleshoot complex technical issues. • Excellent communication and teamwork skills. • Certifications in Databricks or related technologies are a plus.
Key Skills Needed
• pySpark • AWS S3 • Delta lake • Apache Airflow • AWS Glue catalog • AWS Athena • Java programming