Lead Data Engineer

We are looking for a Lead Data Engineer to join our eCommerce Data team. Based from NEXT Head Office in Enderby, Leicestershire!

The Role:

As a Lead Data Engineer, you will play a pivotal role in designing, building and evolving our cloud based data platform on Azure. You will be responsible for developing scalable, reliable and high performance data solutions that power analytics, reporting and emerging AI capabilities across the business.

Working primarily with Azure Databricks, PySpark, SQL and Azure Data Factory, you will lead the development of robust data pipelines, champion engineering best practices and help shape the technical direction of our data platform. You will also drive the design of a modern, AI ready data model that enables faster adoption of machine learning, generative AI and advanced analytics.

This is a hands-on technical leadership role where you'll mentor engineers, collaborate closely with business stakeholders and balance technical excellence with commercial priorities to ensure the team delivers maximum business value.

What You’ll Take On:

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure services such as Azure Data Factory, Azure Databricks, and Azure Data Lake Storage.


• Design and evolve a modern enterprise data model that provides trusted, reusable data assets and accelerates AI, machine learning, and advanced analytics initiatives.


• Ensure data quality, integrity, security, and governance across the data platform by implementing engineering best practices and robust monitoring.


• Collaborate with data scientists, analysts, product teams, and business stakeholders to understand requirements and deliver scalable data solutions.


• Optimise data platforms and pipelines for performance, scalability, reliability, and cost efficiency.


• Lead the implementation of streaming data solutions and Delta Live Tables using PySpark and Databricks.


• Build and maintain CI/CD pipelines for data engineering solutions using Azure DevOps, Git, and infrastructure-as-code where appropriate.


• Integrate with third-party systems and APIs to support secure and reliable data ingestion.


• Mentor and support junior engineers, promoting high engineering standards, knowledge sharing, and continuous improvement.


• Troubleshoot and resolve complex data engineering challenges while ensuring high availability of critical data services.


• Develop a strong understanding of business priorities and commercial objectives, helping to prioritise engineering work that delivers the greatest value and business impact.

What You’ll Bring:

• Strong Python, PySpark and SQL skills with extensive experience designing, developing and optimising large scale data pipelines.


• Hands-on experience with Azure data technologies, including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Delta Lake.


• Experience working with relational and NoSQL databases such as Cosmos DB.


• Experience building streaming data pipelines and Delta Live Tables using Databricks.


• Strong understanding of modern data modelling techniques, including designing data models that support analytics and AI workloads.


• Experience implementing CI/CD pipelines using Azure DevOps, Git, and related tooling.


• Experience integrating with third party APIs and external data sources.


• Strong understanding of data governance, data quality, security and operational best practices.


• Excellent problem solving, communication and stakeholder management skills.


• Strong commercial awareness, with the ability to balance technical priorities against business value and make pragmatic engineering decisions.

Desirable Skills:

• Knowledge of machine learning concepts, MLOps and deploying data products for AI use cases.


• Experience with modern AI and Generative AI platforms, including preparing data for Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) solutions.


• Experience with other cloud platforms such as AWS or GCP.


• Experience with enterprise data governance frameworks and tooling.