M18 - Data Engineer (AWS)

Role Overview

We are looking for experienced Data Engineers to support the development and maintenance of enterprise data products within the organisation's data platform. The role involves designing scalable data solutions, building data pipelines, integrating enterprise systems, and enabling data-driven decision-making across the organisation.

Key Responsibilities

Data Engineering & Platform Integration

• Design, develop, and maintain data pipelines and ETL processes using AWS services such as Glue, Athena, S3, and RDS
• Develop and manage data virtualisation solutions using Denodo and VQL
• Ingest and process data from internal and external data sources
• Perform data extraction, cleansing, transformation, and loading activities
• Implement automated data collection and API integration processes

Data Architecture

• Design and implement conceptual, logical, and physical data models using tools such as ER Studio
• Develop and maintain data warehouses, data lakes, and operational data stores
• Develop and maintain enterprise data blueprints
• Create data marts and analytical views to support business intelligence and reporting needs
• Implement data governance standards and master data management practices

Technical Architecture & Integration

• Ensure seamless integration across various enterprise data systems and applications
• Implement data security, compliance, and governance requirements
• Design scalable solutions for enterprise data integration and consolidation

Development & Analytics

• Develop Python scripts in AWS Glue for data processing and automation
• Write and optimise VQL/SQL queries and stored procedures
• Design and develop RESTful APIs for data services using modern frameworks and best practices
• Work with AWS SageMaker for machine learning model deployment and integration
• Optimise database performance through indexing, query tuning, and maintenance
• Participate in Agile ceremonies including sprint planning, stand-ups, and retrospectives
• Implement and maintain CI/CD pipelines for automated testing and deployment
• Participate in peer code reviews and collaborative development practices

Documentation & Best Practices

• Create and maintain technical documentation for data models, systems, and processes
• Follow coding standards, version control, and change management best practices

Stakeholder Collaboration

• Collaborate with cross-functional teams on data engineering initiatives
• Gather requirements, conduct technical discussions, and implement scalable data solutions
• Work closely with Product Managers, Business Analysts, Data Analysts, Solution Architects, and UX Designers
• Provide technical guidance and support for data-related matters

Requirements

• At least 3 years of experience in Data Engineering or a related role
• Strong proficiency in Python, SQL, and VQL
• Hands-on experience with AWS services such as Glue, Athena, S3, RDS, and SageMaker
• Experience with data virtualisation tools, preferably Denodo
• Familiarity with BI tools such as Tableau or Power BI
• Good understanding of data modelling and database design principles
• Knowledge of data governance and master data management concepts
• Experience with GitLab, version control practices, and CI/CD pipelines
• Experience working in Agile development environments
• Strong analytical, problem-solving, and communication skills
• Ability to work effectively in a collaborative team environment

Nice to Have

• Knowledge of AI technologies such as AWS Bedrock, Azure AI, and Large Language Models (LLMs)