Lead Data Engineer

We have a need for a visionary and technically skilled Lead Data Engineer to lead the design, development, and optimisation of our enterprise data platform. This role is pivotal in shaping our data platform strategy, ensuring scalability, reliability, and performance across our data ecosystem.

Key Accountabilities

• Contribute to the architecture and design of scalable, secure, and high-performance data platforms.


• Lead the design and implementation of the data platform using modern data stack technologies, including but not limited to:
  
  
  
  • Event-driven and batch-based data ingestion and pipelines (e.g., Debezium, Kafka, Flink, Spark)
  
  
  • Data visualisation & reporting (e.g., SSRS, Pentaho)
  
  
  • Data governance (e.g., DataHub)
  
  
  
  


• Collaborate with data architects, principal engineers, analysts, and other engineers to deliver robust data solutions.


• Lead the implementation of automation using tools like Terraform, GitHub Actions, and CI/CD pipelines.


• Ensure data platform reliability, performance, and security.


• Mentor and guide junior engineers while contributing to a culture of technical excellence.


• Evaluate and integrate emerging technologies to enhance platform capabilities.


• Lead incident response and root cause analysis for data platform issues.

Technical Knowledge & Skills

• Expert in GitHub, automation, code/release management, private/public cloud, Docker/Kubernetes.


• Strong knowledge of streaming technologies such as:
  
  
  
  • Debezium CDC
  
  
  • Kafka
  
  
  • Flink
  
  
  • Spark
  
  
  
  


• Experience with ETL/ELT data pipeline development.


• Knowledge of databases (e.g., MSSQL, PostgreSQL).


• Data analytics and reporting (e.g., SSRS, SQL).


• Strong verbal and written communication skills.


• Knowledge of Software Development Life Cycle (SDLC), automation, and governance.


• Proficient in:
  
  
  
  • JIRA
  
  
  • Confluence
  
  
  • Python
  
  
  • Java
  
  
  • .NET
  
  
  
  

Qualifications & Experience

The following qualifications will be highly regarded:

• Hands-on experience with AI and Large Language Models (LLMs).


• Experience working with geographically distributed teams.


• Experience working in a matrixed organisation.