Data Pipeline Engineer

Develop 15 data ingestion pipelines from heterogeneous sources (REST APIs, SFTP file drops, database extracts) → S3 → Glue ETL → Lake Formation

• Implement ETL transformations per R2 data mapping specifications

• Build cross-agency data sharing patterns: Agency B data → Central Platform (Lake Formation cross-account grants, resource links)

• Implement data lineage tagging using OpenLineage / AWS-native lineage metadata for governance audit trail

• Configure data quality checks for multi-source ingestion — handle schema drift, late-arriving data, source unavailability

• Write and maintain IaC (CDK/Terraform) for R2 pipeline resources

• Execute unit testing, integration testing, and cross-agency data access validation

• Support UAT with Agency B data owners — validate data accuracy, timeliness, and access controls

• Document pipeline configurations, source connectivity patterns, and data flow diagrams

• Participate in daily stand-ups, sprint demos, and code reviews

Required skills: AWS Glue, Lake Formation, S3, Athena, Python/PySpark, multi-source integration (APIs, SFTP, DB extracts), IaC, SQL