Data Engineer

🎯 Key Responsibilities

🔹 Data Engineering & Integration

• Design, build, and optimize ETL/ELT pipelines using Apache Spark, PySpark, Databricks, Azure Synapse, or equivalent platforms.
• Develop scalable batch and real-time data processing solutions.
• Integrate data from Core Banking, Payments, Treasury, Trade Finance, CRM, Compliance, and Risk systems.
• Develop and maintain enterprise data models including 3NF, Dimensional Modeling, and Data Vault 2.0.

🔹 Streaming & Modern Data Platforms

• Build and operationalize real-time streaming pipelines using Kafka, Confluent, or Azure Event Hubs.
• Support data platform modernization initiatives, including migration from legacy platforms (e.g., Teradata, DB2) to cloud-native environments such as Snowflake, Databricks, or Azure Synapse.
• Implement scalable cloud-based data lake and data warehouse architectures.

🔹 Data Quality & Governance

• Implement data quality, validation, lineage, and observability frameworks using tools such as Great Expectations, Deequ, or dbt.
• Collaborate with Governance and Security teams to ensure compliance with enterprise data standards.
• Support metadata management, cataloging, and lineage initiatives using Azure Purview, Apache Atlas, or Collibra.

🔹 Regulatory & Compliance Support

• Support regulatory reporting and risk data flows including:MAS 610MAS 649Basel III / Basel IVIFRS 9 / IFRS 17BCBS 239
• Ensure data security controls including encryption, tokenization, masking, RBAC, and audit logging are implemented.

🔹 DevOps & MLOps

• Develop CI/CD pipelines using Azure DevOps, GitHub Actions, or Terraform.
• Collaborate with Data Scientists and AI teams to deploy ML feature stores and model-serving pipelines.
• Support automation and Infrastructure-as-Code (IaC) initiatives.

🛠 Required Technical Skills

Programming Languages

• Python
• PySpark
• SQL
• Scala

Data Platforms

• Azure Data Lake
• Azure Synapse Analytics
• Databricks
• Snowflake

Data Orchestration

• Apache Airflow
• Azure Data Factory (ADF)
• dbt

Streaming Technologies

• Apache Kafka
• Confluent Platform
• Azure Event Hubs

Data Governance

• Azure Purview
• Apache Atlas
• Collibra

Security & Compliance

• Encryption
• Tokenization
• Role-Based Access Control (RBAC)
• Audit Logging

DevOps & Infrastructure

• Terraform
• Azure DevOps
• GitHub Actions

🎓 Qualifications

• Bachelor's or Master's Degree in Computer Science, Data Engineering, Information Technology, or related discipline.
• 6–10 years of Data Engineering experience.
• Minimum 3 years of experience within Banking, Financial Services, Insurance, or Capital Markets environments.
• Strong experience designing and implementing cloud-based data platforms on Azure and/or AWS.
• Hands-on experience with batch and real-time data processing frameworks.
• Understanding of regulatory reporting and risk data management frameworks.