Senior Data Engineer - Part-Time

Responsibility

Data Pipeline Development and Architecture

a) Translate data requirements from business users into technical specifications.

b) Build out data products as part of a data team.

c) Architect and build ingestion pipelines to collect, clean, merge, and harmonise

data from different source systems.

d) Construct, test, and update useful and reusable data models based on data

needs of end users.

e) Design and build secure mechanisms for end users and systems to access data

in the data warehouse.

f) Research, propose, and develop new technologies and processes to improve

agency data infrastructure.

Data Quality, Testing and Validation

a) Implement data quality checks and validation processes to ensure data accuracy

and consistency across all pipeline stages.

b) Design and execute test cases for data pipelines, covering functional, regression,

and exploratory testing to validate data transformations and business logic.

c) Develop and maintain automated test coverage for data pipelines to prevent bugs

and defects from reaching production.

d) Perform data reconciliation and integrity testing to validate end-to-end data flows

across source and target systems.

e) Support User Acceptance Testing (UAT) by coordinating with stakeholders to

validate that data products and pipelines meet business requirements.

Operations, Governance and Continuous Improvement

a) Conduct day-to-day monitoring of databases and ETL systems, including capacity

planning, maintenance, performance tuning, and issue diagnosis.

b) Collaborate with partner agency IT teams on technology stack, infrastructure, and

security alignment.

c) Collaborate with data stewards to establish and enforce data governance policies,

best practices, and procedures.

d) Maintain a data catalogue to document data assets, metadata, and lineage.

e) Implement and enforce data security best practices, including access control,

encryption, and data masking, to safeguard sensitive data.

f) Drive continuous improvement in CI/CD and Site Reliability Engineering (SRE)

practices.

Experience and Skills Needed

Qualifications and Experience

a) Bachelor's Degree, preferably in Computer Science, Software Engineering,

Information Technology, or related disciplines.

b) Deep understanding of system design, data structures and algorithms, data

modelling, data access, and data storage.

c) Experience in designing, building, and maintaining batch and real-time data

pipelines.

d) Experience with orchestration frameworks such as Airflow and Azure Data

Factory.

e) Proficiency in working with Python, Shell Scripts, and SQL.

f) Demonstrated ability in using cloud technologies such as AWS, Azure, and Google Cloud.

g) Experience with Databricks.

Data Quality and Testing

a) Experience designing and executing data quality checks, validation processes,

and test cases for data pipelines.

b) Familiarity with test automation approaches for data pipelines, including

integration with CI/CD pipelines.

c) Experience with defect tracking and test management tools such as JIRA or

equivalent.

DevOps and CI/CD Practices

a) Familiarity with building and using CI/CD pipelines.

b) Familiarity with DevOps tools such as Docker, Git, and Terraform.

c) Experience implementing technical processes to enforce data security, data

quality, and data governance.

Domain Knowledge (Added Advantage)

a) Familiarity with government systems and policies relating to data governance,

data management, data infrastructure, and data security.

b) Experience in Climate and Weather domains will be an advantage.