Senior Data Engineer

• Data Warehouse, Lake and Lakehouse architecture patterns.


• Distributed data processing using frameworks such as Spark or Flink.


• Designing and supporting batch and near-real-time ingestion pipelines.


• Building incremental, idempotent, and fault-tolerant data pipelines.


• Data quality, reconciliation, and observability practices.


• Metadata, lineage, governance, and access control concepts.


• Analytical data modelling and efficient data structures for warehouse and large-scale query workloads.


• Medallion architecture such as Bronze, Silver, and Gold layers.


• Open table formats such as Iceberg.


• Schema evolution, partitioning strategies, file optimisation, and storage layout tuning.


• Event-driven or streaming platforms such as Kafka, Pulsar, or Redpanda.


• Columnar or high-performance analytical platforms such as ClickHouse.


• CI/CD pipelines, deployment automation, and engineering standards for data workloads.


• Experience improving reliability, performance, and scalability across production data platforms.


• Familiarity with monitoring, alerting, observability, and operational support practices.


• Exposure to containerised or clustered environments such as Kubernetes, OpenShift, or similar platforms.


• Strong debugging capability across data, pipeline, compute, and platform layers.


• Strong sense of ownership and accountability.


• Comfortable making technical trade-offs while remaining pragmatic and hands-on.


• Excellent problem-solving and communication skills.


• Able to collaborate effectively with technical and non-technical stakeholders.


• Passionate about building scalable, maintainable, and well-documented systems.


• Committed to sharing knowledge and raising engineering standards across the team.

Data Platform Engineering:

• Design, build, and improve critical batch and near-real-time data pipelines that support enterprise analytics and operational use cases.


• Develop reusable engineering patterns for ingestion, transformation, storage, and serving layers across the platform.


• Own important platform components and ensure they are reliable, scalable, and supportable in production.


• Contribute directly to the modernisation of legacy data workflows into stronger platform-aligned solutions.

Lakehouse Architecture & Scalable Processing:

• Contribute to the design and evolution of the enterprise data lake or lakehouse platform.


• Implement and refine engineering standards for storage layout, transformation patterns, and data processing frameworks.


• Optimise partitioning, schema evolution, and file organisation to improve performance and maintainability.


• Build and support distributed data processing solutions using modern frameworks and platform tooling.

Data Quality, Reliability & Governance:

• Design and implement data quality, reconciliation, and observability controls for critical datasets and platform flows.


• Ensure key pipelines and datasets meet expectations for freshness, completeness, accuracy, and recoverability.


• Strengthen metadata, lineage, and documentation practices across the platform.


• Work with governance and security stakeholders to support compliant, well-controlled data management practices.

Technical Leadership & Collaboration:

• Partner with BI, analytics, software engineering, product, and business stakeholders to understand and support data use cases.


• Translate business and platform requirements into scalable, well-engineered technical solutions.


• Provide hands-on technical leadership during planning, design, implementation, and operational improvement.


• Mentor intermediate and junior data engineers through code reviews, design guidance, and practical knowledge-sharing.

Continuous Improvement & Innovation:

• Identify opportunities to improve platform performance, resilience, scalability, and cost efficiency.


• Drive automation, standardisation, and maintainability across the data platform.


• Evaluate new tools or patterns where they provide clear value to the platform or team.


• Help grow the maturity of BET Software's data engineering capability over time.

Tech Environment:

• The platform may include a combination of established and modern technologies such as: SQL Server, Python, Spark, Flink, Airflow, Object storage, Open format Tables, Kafka / Redpanda, ClickHouse or similar columnar analytical stores, Git and CI/CD tooling, Kubernetes, OpenShift, or similar runtime environments.

• Degree or diploma in IT, Computer Science, Engineering, or a related technical discipline.


• 6+ years of experience in data engineering, ETL/ELT development, or data platform engineering.


• Strong hands-on SQL expertise, including advanced performance tuning, query optimisation, indexing strategies, and efficient analytical data design.


• Proven experience building and operating modern data platform components in production environments.


• Strong experience working with object storage in cloud or on-premises environments.


• Experience with workflow orchestration platforms such as Airflow, SQL Server Agent, or similar.

Technical & Architectural Skills:

• Data Warehouse, Lake and Lakehouse architecture patterns.


• Distributed data processing using frameworks such as Spark or Flink.


• Designing and supporting batch and near-real-time ingestion pipelines.


• Building incremental, idempotent, and fault-tolerant data pipelines.


• Data quality, reconciliation, and observability practices.


• Metadata, lineage, governance, and access control concepts.


• Analytical data modelling and efficient data structures for warehouse and large-scale query workloads.

Experience in the following areas is highly valuable:

• Medallion architecture such as Bronze, Silver, and Gold layers.


• Open table formats such as Iceberg.


• Schema evolution, partitioning strategies, file optimisation, and storage layout tuning.


• Event-driven or streaming platforms such as Kafka, Pulsar, or Redpanda.


• Columnar or high-performance analytical platforms such as ClickHouse.


• CI/CD pipelines, deployment automation, and engineering standards for data workloads.

Platform & Engineering Practices:

• Experience improving reliability, performance, and scalability across production data platforms.


• Familiarity with monitoring, alerting, observability, and operational support practices.


• Exposure to containerised or clustered environments such as Kubernetes, OpenShift, or similar platforms.


• Strong debugging capability across data, pipeline, compute, and platform layers.

Personal Attributes:

• Strong sense of ownership and accountability.


• Comfortable making technical trade-offs while remaining pragmatic and hands-on.


• Excellent problem-solving and communication skills.


• Able to collaborate effectively with technical and non-technical stakeholders.


• Passionate about building scalable, maintainable, and well-documented systems.


• Committed to sharing knowledge and raising engineering standards across the team.

Living the Spirit 

• Engages in cross-functional collaboration and problem solving while contributing to an inclusive team culture.


• Supports a culture of adaptability, accountability, and shared success across the department and wider business.


• Shows up authentically and contributes to team outcomes by working effectively with diverse colleagues and perspectives.


• Approaches challenges as opportunities to learn, improve, and help others grow.