Data Engineer

Opportunity 

What you'll do:

• Design and evolve our data architecture leveraging Apache Iceberg on S3 and Snowflake, balancing performance, reliability, and cost.


• Lead continuous improvement of schemas, data models, pipelines, and engineering standards, own design docs and review forums.


• Plan and coordinate data migrations with zero/low downtime patterns, including backfills, cutovers, and data validation.


• Standardize data contracts and enforce quality checks throughout pipelines and transformations.


• Implement and operate Iceberg tables: catalog strategy (AWS Glue), partition transforms, schema evolution, and time-travel/snapshot management.


• Optimize data layout (partitioning, clustering, file sizing, compaction) to improve read/write performance and control costs across engines.


• Build and maintain batch and streaming pipelines using Airflow, AWS Glue, Step Functions, Lambda, Kinesis, and Snowflake.


• Design normalized and dimensional models; apply partitioning and clustering strategies appropriate to Iceberg and target engines.


• Own SQL and Spark performance tuning, job optimization, and cost governance (e.g., Snowflake warehouse sizing, query profile analysis).


• Establish SLAs, lineage, tests, alerts, and runtime metrics; integrate data quality checks into CI/CD and orchestration.


• Mentor and pair program; elevate craftsmanship, testing, reliability, and operational excellence.


• Promote security best practices, governance, and compliance-by-default patterns for sensitive data.


• Provide documentation, code examples, and training that enable partners to self-serve; champion code reviews and design best practices.

Who you are:

• 4+ years of professional data engineering experience delivering production-grade pipelines and data platforms.


• Strong problem-solving and analytical skills; track record of decomposing complex problems and shipping pragmatic solutions.


• Excellent communication and documentation; teacher/mentor who raises the bar for the team.


• Proficiency in Python and SQL; experience with Java or Scala a plus.


• Solid software engineering practices: Git/GitHub, branching, code review, testing (Pytest/JUnit), and CI/CD.


• Hands-on experience operating Apache Iceberg on AWS (S3 + Glue); deep familiarity with Parquet/ORC.


• Operating Iceberg with Spark (PySpark/Spark SQL); exposure to Snowflake Iceberg Tables and/or Trino/Presto is a plus.


• Practical knowledge of Iceberg concepts: metadata/manifest layers, partition transforms, schema evolution, snapshots/time-travel/branching, row-level deletes/merges, compaction, and vacuum/snapshot expiration.


• AWS expertise: S3, Lambda, EventBridge, Glue, Athena, EMR, Kinesis, Step Functions, SQS; cost/performance governance across services.


• Snowflake proficiency; experience exposing Iceberg data via Snowflake external or Iceberg tables; Redshift/BigQuery a plus.


• Strong SQL performance tuning across at least one engine (Snowflake, PostgreSQL/RDS, MySQL).


• Observability with Datadog and/or CloudWatch; strong log analysis, performance profiling, and incident response.


• Practical use of AI coding assistants (GitHub Copilot, Amazon Q Developer, ChatGPT, Claude) (nice to have

#LI-AD1