Requirements
Key Responsibilities:
Data Engineering and Platform Delivery:
• Design, develop, test, deploy, and maintain scalable data pipelines using modern cloud -native and enterprise data engineering tools.
• Build robust ETL/ELT processes to ingest, transform, validate, and publish data from multiple structured and unstructured sources.
• Work with batch, near -real -time, and streaming data processing patterns where required.
• Develop reusable data engineering components, frameworks, templates, and automation scripts.
• Support the development of data lakes, lakehouses, data warehouses, operational data stores, and analytics platforms.
• Optimise data pipelines for performance, cost, reliability, scalability, and maintainability.
• Ensure data engineering solutions are production -ready, supportable, monitored, and documented.
Cloud and Technology Implementation:
• Build data solutions on cloud platforms such as Microsoft Azure, AWS, or Google Cloud, with strong preference for Azure experience.
• Work with technologies such as AWS Glue, Azure Data Factory, Synapse Analytics, Databricks, Fabric, Data Lake Storage, SQL, Python, Spark, Power BI, Snowflake, dbt, Airflow, Kafka, or equivalent tooling.
• Implement data ingestion from APIs, databases, files, SaaS platforms, event streams, and third -party systems.
• Use infrastructure -as -code, CI/CD pipelines, and automated deployment approaches where appropriate.
• Collaborate with DevOps and platform teams to ensure secure and reliable deployment of data workloads.
Data Modelling, Quality, and Governance:
• Design and implement appropriate data models, including dimensional models, data vault, star schemas, and curated analytical datasets.
• Apply data quality rules, validation checks, reconciliation controls, and exception handling.
• Support metadata management, lineage, data cataloguing, and governance requirements.
• Ensure solutions comply with data security, privacy, access control, retention, and audit requirements.
• Work with business and technical stakeholders to define data definitions, mapping rules, transformation logic, and acceptance criteria.
Technical Leadership:
• Lead data engineering workstreams from discovery through to design, build, test, deployment, and support transition.
• Provide technical guidance, mentoring, and code reviews for junior and mid -level data engineers.
• Translate high -level architecture into practical engineering designs and delivery tasks.
• Contribute to technical decision -making, estimation, planning, and risk management.
• Identify engineering risks, dependencies, blockers, and improvement opportunities early.
• Promote engineering standards, reusable patterns, documentation, and good development practices.
Stakeholder and Delivery Management:
• Work closely with product owners, business analysts, architects, testers, data analysts, and client stakeholders.
• Participate in agile ceremonies including sprint planning, daily stand -ups, backlog refinement, reviews, and retrospectives.
• Support discovery workshops, requirements analysis, technical design sessions, and show -and -tell demonstrations.
• Produce clear technical documentation, data flow diagrams, mapping specifications, deployment guides, and support documentation.
• Support transition into live service, including knowledge transfer, runbooks, monitoring, incident response, and handover to support teams.
Required Skills and Experience
Essential Technical Skills
• Strong experience as a Data Engineer or Senior Data Engineer in enterprise or cloud environments.
• Strong SQL skills, including query optimisation, stored procedures, data modelling, and performance tuning.
• Strong Python or PySpark experience for data processing, automation, and transformation logic.
• Experience building ETL/ELT pipelines using tools such as AWS Glue, Azure Data Factory, Databricks, Synapse, Fabric, dbt, Airflow, Informatica, Talend, or similar.
• Experience working with cloud data platforms, preferably Microsoft Azure.
• Experience with data lake, lakehouse, data warehouse, and analytical platform architectures.
• Good understanding of batch processing, incremental loads, CDC, API ingestion, and file -based ingestion patterns.
• Experience with data validation, reconciliation, error handling, and data quality controls.
• Experience using Git -based source control and CI/CD practices.
• Understanding of security, access control, encryption, data privacy, and environment management.
Essential Delivery Experience:
• Experience delivering production -grade data platforms or pipelines in complex organisations.
• Ability to work across the full delivery lifecycle from requirements and design through to build, test, release, and support.
• Experience working in agile delivery teams.
• Ability to produce clear technical documentation and explain technical concepts to non -technical stakeholders.
• Experience leading technical workstreams or mentoring other engineers.
• Strong analytical, problem -solving, and troubleshooting skills.
• Ability to work independently, manage priorities, and take ownership of outcomes.
Desirable Skills and Experience:
• Microsoft Azure certifications, such as Azure Data Engineer Associate or equivalent.
• Experience with AWS Glue, Microsoft Fabric, Azure Synapse Analytics, Azure Data Lake, Azure SQL, Azure Functions, Logic Apps, Event Hubs, or Azure Purview.
• Experience with Databricks, Delta Lake, Spark, Unity Catalog, MLflow, or lakehouse patterns.
• Experience with Snowflake, Redshift, BigQuery, or other cloud data warehouse platforms.
• Experience with dbt, data transformation frameworks, or analytics engineering practices.
• Experience with streaming technologies such as Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Experience with Power BI semantic models, reporting datasets, or analytical consumption layers.
• Experience with data governance, data lineage, metadata management, master data management, or data cataloguing.
• Experience with Terraform, Bicep, ARM templates, Docker, Kubernetes, or other infrastructure and deployment tooling.
Behavioural Competencies:
• Strong ownership mindset with the ability to take accountability for technical delivery.
• Clear and confident communicator, able to engage with technical and business stakeholders.
• Pragmatic problem solver who balances engineering quality with delivery timelines.
• Collaborative team player who supports others and contributes to shared outcomes.
• Detail -oriented, with strong focus on data accuracy, quality, and operational reliability.
• Comfortable working in fast -paced, multi -disciplinary, and multi -supplier environments.
• Able to challenge constructively and recommend practical improvements.
• Committed to continuous learning and keeping up to date with modern data engineering practices.
Typical Deliverables:
• Data pipeline designs and implemented ETL/ELT workflows.
• Data ingestion, transformation, validation, and publishing components.
• Data models, schemas, mapping documents, and transformation specifications.
• Automated deployment pipelines and environment configuration.
• Data quality checks, reconciliation reports, and exception handling processes.
• Technical design documentation and data flow diagrams.
• Runbooks, operational guides, and support handover documentation.
• Performance optimisation recommendations and implemented improvements.
• Knowledge transfer sessions and mentoring for internal teams.
Qualifications:
• Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, Mathematics, Statistics, or a related discipline, or equivalent professional experience.
• Relevant cloud or data engineering certifications are desirable but not mandatory.
Experience Level:
• 5+ years of experience in data engineering, software engineering, or data platform delivery.
• At least 2+ years of hands -on experience delivering cloud -based data engineering solutions.
• Prior experience in a senior, lead, or workstream ownership role is preferred.
Your email job alert is now active!