Sr Data Engineer

Responsibilities

• Design and implement modern data ingestion pipelines for business systems data using Fivetran, Python and Bulk APIs, S3-based lakehouse architectures, and streaming ingestion (Kafka, Spark)


• Build end-to-end automated pipelines from source systems to Snowflake, ensuring reliability, scalability, and observability


• Apply advanced SQL in Snowflake for transformation, standardization, and preparation of analytics-ready datasets


• Optimize ingestion performance for large-scale, high-volume datasets using API rate-limit handling, micro-batching, and incremental loading / CDC


• Implement DevOps and CI/CD practices for data pipelines, including version control and deployment automation


• Partner with analysts, SMEs, and platform teams to rapidly prototype pipelines supporting exploration and evolving requirements


• Monitor, troubleshoot, and optimize ingestion jobs to ensure data freshness, accuracy, and performance

• 6+ years of hands-on experience in data engineering and ingestion automation in cloud or hybrid environments


• Proven experience ingesting business application data (CRM, ERP, financial, consumption, renewals systems)


• Advanced proficiency with Python and Fivetran (or similar ETL/ELT tool) for pipelines, automation, orchestration, and validation


• Expertise with Kafka, Spark, and S3 / lakehouse-style storage architectures


• Strong proficiency with Snowflake, ingestion, transformation, and performance optimization


• Deep understanding of ingestion optimization, rate limits, micro-batching, incremental loads, and CDC


• Experience applying DevOps / CI/CD practices to data pipelines


• Ability to work independently and deliver solutions rapidly in a fast-paced, iterative environment

     Desirable:

• Experience with additional open-source ingestion frameworks and connectors


• Exposure to event-driven or telemetry-heavy architectures at scale