Data Engineer

We are seeking a highly skilled Data Engineer to design, build, and optimize scalable data ingestion and transformation pipelines. The ideal candidate will be responsible for developing end-to-end data pipelines, implementing data quality frameworks, and enabling reliable data delivery into a governed data lake environment.

Responsibility

• Design and implement ETL transformations based on defined data mapping specifications.
• Build and maintain data quality validation frameworks to ensure completeness, schema conformance, and referential integrity.
• Configure error handling, exception management, and dead-letter queue patterns for failed ingestion records.
• Develop and maintain Infrastructure as Code (IaC) using Terraform or AWS CDK for data platform resources.
• Execute unit testing and integration testing to validate pipeline functionality and data accuracy.
• Support User Acceptance Testing (UAT) by validating output datasets, schemas, and record counts with business stakeholders.
• Create and maintain technical documentation including pipeline configurations, runbooks, and data flow diagrams.
• Participate in Agile ceremonies including sprint planning, stand-ups, demos, and code reviews.

Requirement

• 5+ years of experience in Data Engineering or Data Platform Development.
• Strong hands-on experience with AWS Glue, AWS Lake Formation, Amazon S3, and Amazon Athena.
• Proficiency in Python, PySpark, and SQL.
• Experience building ETL/ELT pipelines in cloud-native environments.
• Hands-on experience with Infrastructure as Code tools such as Terraform or AWS CDK.
• Strong understanding of data modeling, data governance, and metadata management.
• Experience with automated testing and deployment practices.