Senior Data Engineer (Databricks)

About the Role

We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that enable data-driven decision-making. You will work closely with our client's stakeholders to develop reliable, secure, and high-performance data solutions while driving best practices in Data Engineering, CI/CD, and Site Reliability Engineering (SRE).

This role is ideal for engineers who are passionate about building modern data platforms, ensuring data quality, and delivering resilient cloud-based data solutions.

Key Responsibilities

Data Pipeline Development & Architecture

• Translate business data requirements into technical solutions.
• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build data ingestion pipelines to collect, cleanse, transform, and integrate data from multiple source systems.
• Design reusable, scalable, and optimized data models to support analytics and reporting.
• Develop secure mechanisms for users and applications to access enterprise data.
• Research, evaluate, and implement new technologies to improve the organization's data infrastructure.

Data Quality, Testing & Validation

• Design and implement data quality frameworks to ensure data accuracy, completeness, and consistency.
• Develop and execute functional, regression, and exploratory testing for data pipelines.
• Build automated testing capabilities to improve deployment quality and reduce production defects.
• Perform data reconciliation and end-to-end validation across multiple systems.
• Support User Acceptance Testing (UAT) and collaborate with business stakeholders to validate solutions.

Operations, Governance & Continuous Improvement

• Monitor and maintain databases and ETL platforms, including performance tuning, capacity planning, troubleshooting, and issue resolution.
• Collaborate with infrastructure and security teams to ensure scalable and secure data platforms.
• Work with data stewards to establish and enforce data governance standards and best practices.
• Maintain metadata repositories and data catalogues.
• Implement data security controls, including access management, encryption, and data masking.
• Drive continuous improvement in CI/CD pipelines, automation, and Site Reliability Engineering (SRE) practices.

Requirements

Qualifications

• Bachelor's Degree in Computer Science, Information Technology, Software Engineering, or a related discipline.
• Minimum 5 years of experience in Data Engineering or related roles.

Technical Skills

• Strong understanding of Data architecture Data modelling Data structures and algorithms Database design and data storage concepts
• Hands-on experience designing and maintaining batch and streaming data pipelines.
• Strong programming skills in Python SQL Shell Scripting
• Experience with workflow orchestration tools such as Apache Airflow Azure Data Factory
• Hands-on experience with Databricks.
• Experience working with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP).

Data Quality & Testing

• Experience implementing data quality frameworks and validation processes.
• Experience developing automated testing for data pipelines.
• Familiarity with defect tracking and test management tools such as JIRA.

DevOps & CI/CD

• Experience building and maintaining CI/CD pipelines.
• Familiarity with Docker Git Terraform
• Understanding of infrastructure automation and deployment best practices.
• Experience implementing data governance, security, and compliance standards.

Preferred Qualifications

Candidates with the following experience will have an advantage:

• Government sector projects.
• Data governance, data infrastructure, or data security initiatives.
• Climate or Weather domain projects.

Required Technical Skills

• Python
• SQL
• Shell Scripting
• Apache Airflow
• Azure Data Factory
• Databricks
• AWS / Azure / Google Cloud Platform (GCP)
• ETL Development
• Data Pipelines
• Data Modelling
• Data Quality Testing
• Data Governance
• CI/CD
• Docker
• Git
• Terraform
• Site Reliability Engineering (SRE)
• JIRA