Data Engineer

Data Engineer [for Project Domino; preferably senior 5 years and up experience]

Responsibilities

Design, architect and maintain secure data platforms such as setting up data storage solutions like SQL databases

Implement and maintain optimized workflows, ETL / ELT processes, and pipelines; deliver flexibility for data consumption by designing batch and streaming data use cases

Ensure support for traceability and reproducibility through enabling of data lineage features for ingested data

Work with colleagues such as Data Scientists, Data Analysts, software and data engineers, and architects on implementing solutions involving data on distributed systems and platforms

Create highly available and scalable capabilities that deliver data related tooling and manage data platforms and pipelines for stakeholders

Qualifications

Proficiency and experience in software engineering programming languages such as Python. Experience in Golang is preferred

Expertise in SQL / RDBMS databases and proficiency in setting up these resources. Additional experience in NoSQL databases and setting up these is preferred

Extensive experience in design and implementation of data handling processes and pipelines (ingestion, transformation, modelling, and storage) using Python, Apache Airflow, Spark

Expertise in cloud platforms and technologies (Azure, AWS, GCP) is a plus but not required

Experience with streaming platform setup and consumption such as Kafka, Flink

Experience with consumption from other storages for streaming such as S3, Spark

Experience in containers and container orchestration platforms such as Docker and Kubernetes

Experience in CI / CD and Devops, scripting languages such as shell and automation

Preferably a degree in the related fields of Computer Science

Strong problem solving and solutions engineering mindset

Experience in working and coordinating remotely