Lead Data Engineer

                                                                            Job Description – Lead Data Engineer

Data Engineer (Azure | Databricks | PySpark)
Permanent | Hybrid | UK Public Sector Project

OVERViEW

We are seeking a hands-on Data Engineer to join a growing Data & Analytics practice. This role focuses on building, optimising, and maintaining modern cloud data platforms.

You will work across the full data lifecycle, delivering scalable, high-performance solutions using Azure and Databricks.

Who We’re Looking For

A strong Data Engineer with deep technical expertise in Azure and Databricks, who enjoys building robust pipelines and working in a collaborative, client-facing environment.

Key Responsibilities

• Build and maintain scalable data pipelines (batch & real-time)


• Develop solutions using Azure Databricks and PySpark


• Implement Medallion Architecture (Bronze, Silver, Gold)


• Design ETL/ELT pipelines using Azure Data Factory


• Develop data warehouse and lakehouse solutions


• Support BI, analytics, and AI workloads


• Optimise performance, cost, and reliability


• Implement CI/CD pipelines using Azure DevOps


• Collaborate with architects, analysts, and stakeholders

Mandatory Requirements

• Strong hands-on experience with Azure Databricks (PySpark)


• Experience with Azure Data Factory and Synapse


• Strong SQL and data warehousing expertise


• Experience with data modelling (star, snowflake, Kimball)


• Experience with Power BI and Microsoft Fabric


• Strong programming skills (Python, SQL, scripting)


• Experience with CI/CD and DevOps practices


• Ability to produce technical documentation (HLD, LLD)

Desirable

• Experience in consulting / professional services


• Exposure to AI and advanced analytics solutions


• Experience with real-time / streaming data


• Experience in regulated industries

Soft Skills

• Strong problem-solving skills


• Collaborative and team-oriented mindset


• Clear communication and documentation skills


• Ability to work in fast-paced delivery environments

Contract Details

• Role Type: Permanent & Hybrid (2 days per week in office)


• Location: Guildford


• Clearance: Candidates must be eligible for UK Security Clearance (SC) or active SC