Associate Manager, D&T, Data Engineering

Design and maintain scalable batch and real-time data pipelines on Databricks using PySpark, Structured Streaming, DBT, and Delta Live Tables. Manage cloud infrastructure with Terraform on AWS and deploy workflows using Databricks Asset Bundles with CI/CD best practices. Collaborate with Data Scientists to produce ML models using MLflow and ensure data quality through automated testing and robust deployment pipelines. Design, build, and maintain scalable batch and real-time data pipelines using PySpark and Structured Streaming on Databricks, including migration of legacy ETL processes to Delta Live Tables (DLT). Develop and manage robust transformation layers using DBT to deliver clean, tested, and well-documented data models. Implement Infrastructure as Code (IaC) using Terraform to provision and manage AWS resources (S3, IAM, Glue) and Databricks workspaces. Deploy and manage data workflows using Databricks Asset Bundles (DAB) with standardized CI/CD practices via GitHub Actions or similar tools. Optimize Spark workloads for performance and cost efficiency, leveraging Photon Engine, Auto Loader, and best-practice tuning techniques. Enforce data governance and quality standards using Unity Catalog (row-level security) and support MLOps initiatives through MLflow-based model tracking and production deployment.