Senior Data Engineer

Role Overview

We are seeking a Data Engineer to design, develop, and maintain scalable, secure, and reliable data platforms and pipelines that support business and operational needs. The successful candidate will work closely with stakeholders, engineers, and data stewards to deliver high-quality data products while ensuring strong governance, security, and operational excellence.

Key Responsibilities

Data Pipeline Development and Architecture

• Translate business and data requirements into technical specifications and scalable data solutions.

• Develop and maintain data products as part of a cross-functional data team.

• Design, build, and optimise data ingestion pipelines to collect, transform, integrate, and harmonise data from multiple source systems.

• Design, develop, test, and maintain reusable data models to support reporting, analytics, and operational use cases.

• Implement secure and efficient mechanisms for users and applications to access enterprise data assets.

• Evaluate and implement new technologies, tools, and processes to enhance data infrastructure capabilities and performance.

Data Quality, Testing and Validation

• Develop and implement data quality controls and validation processes to ensure data accuracy, completeness, and consistency.

• Design, execute, and maintain test cases for data pipelines, including functional, regression, and exploratory testing.

• Build and maintain automated testing frameworks to improve data reliability and reduce production defects.

• Perform data reconciliation and integrity testing to validate end-to-end data processing and transformation workflows.

• Support User Acceptance Testing (UAT) activities by working with stakeholders to validate business requirements and expected outcomes.

Operations, Governance and Continuous Improvement

• Monitor and maintain databases, ETL processes, and data platforms, including performance optimisation, troubleshooting, capacity planning, and operational support.

• Collaborate with technology teams and stakeholders to align infrastructure, security, and platform requirements.

• Support the implementation and enforcement of data governance policies, standards, and best practices.

• Maintain data catalogues, metadata repositories, and data lineage documentation.

• Implement data security controls, including access management, encryption, masking, and other data protection measures.

• Drive continuous improvements in CI/CD, DevOps, and Site Reliability Engineering (SRE) practices.

Requirements

Qualifications and Experience

• Bachelor's Degree in Computer Science, Software Engineering, Information Technology, or a related discipline.

• Strong understanding of system design principles, data structures and algorithms, data modelling, data access patterns, and data storage technologies.

• Experience designing, building, and maintaining batch and real-time data pipelines.

• Hands-on experience with workflow orchestration tools such as Apache Airflow and Azure Data Factory.

• Proficiency in Python, SQL, and Shell scripting.

• Experience working with cloud platforms such as AWS, Microsoft Azure, and Google Cloud Platform (GCP).

• Experience with Databricks.

• Experience implementing data quality frameworks, validation processes, and testing methodologies for data pipelines.

• Familiarity with automated testing approaches and integration within CI/CD environments.

• Experience with issue tracking and test management tools such as JIRA or equivalent platforms.

• Familiarity with CI/CD pipelines and DevOps practices.

• Experience using DevOps tools such as Docker, Git, and Terraform.

• Experience implementing solutions that support data security, data quality, and data governance requirements.

Preferred Experience

• Familiarity with government-sector data management, governance, infrastructure, and security frameworks.

• Experience in climate, weather, environmental, or geospatial data domains will be advantageous.

Additional Information

• Appointment level and remuneration will be commensurate with qualifications, skills, and relevant experience.