FDE - DATA

Develop 20 data table ingestion pipelines end-to-end (source extraction → S3 landing → Glue ETL → Lake Formation curated zone)

Implement ETL transformations per data mapping specifications

Build data quality validation rules (Great Expectations / custom Glue checks) — completeness, schema conformance, referential integrity

Configure error handling and dead-letter patterns for failed ingestion records

Register all datasets in AWS Glue Data Catalog with standardised metadata tags (owner, classification, freshness SLA)

Write and maintain IaC (CDK/Terraform) for pipeline resources — Glue jobs, crawlers, S3 buckets, IAM roles

Execute unit testing (per-transform logic) and integration testing (end-to-end flow with sample data)

Support UAT with Agency A data owners — validate output tables match expected schema and row counts

Document pipeline configurations, runbooks, and data flow diagrams for handover

Participate in daily stand-ups, sprint demos, and code reviews

What We Are Looking For

Required skills: AWS Glue, Lake Formation, S3, Athena, Python/PySpark, IaC (Terraform/CDK), SQL

Nice to Have

Great Expectations, SHIP-HATS CI/CD, data cataloging, prior public sector / Government Agency experience