Tech Lead Databricks / Data Platform

Role: Tech Lead – Databricks / Data
Platform
Location: Remote
Employment Type: Full-Time
Experience: 10–12 Years

JOB
SUMMARY

We
are looking for an experienced Tech Lead – Databricks / Data Platform to
lead the design, development, and optimization of a scalable, secure, and
reliable cloud-based data platform. The role will provide technical leadership
across Databricks, Azure, CI/CD, Infrastructure as Code, orchestration, data
security, FinOps, and platform automation.

The
ideal candidate will be hands-on, technically strong, and experienced in
leading engineering teams while partnering closely with Data Engineers,
Analytics Engineers, Architects, Security, and Cloud teams to deliver
enterprise-grade data solutions.

KEY
RESPONSIBILITIES

• Lead
  the architecture, design, and implementation of Databricks-based data
  platforms and cloud-native data pipelines.
  
• Design
  and implement robust CI/CD pipelines for data ingestion,
  transformation, dbt, SQL, notebooks, and Databricks workloads.
  
• Lead
  end-to-end orchestration of Databricks Jobs and Workflows,
  including ingestion, transformation, data quality checks, and
  dependencies.
  
• Define
  and implement automated testing and quality gates within CI/CD
  pipelines, including schema, data model, and contract validation.
  
• Drive Infrastructure
  as Code (IaC) practices using Terraform and reusable modules for
  Databricks and Azure infrastructure.
  
• Automate
  Databricks platform operations, including workspace and cluster
  provisioning, runtime and library management, job deployment,
  configuration, and environment management.
  
• Establish
  and maintain Identity and Access Management (IAM) across Databricks
  and Azure, including RBAC, TBAC, service principals, groups, roles, and
  workspace/cluster/table-level access controls.
  
• Work
  closely with the EDP Architect and Security teams to implement secure and
  scalable Unity Catalog and data governance patterns.
  
• Develop
  reusable Terraform modules, Databricks job templates, Airflow DAG
  patterns, and engineering frameworks to accelerate onboarding of new
  projects.
  
• Define
  and govern the code promotion and release management process across
  development, QA, and production environments.
  
• Lead
  end-to-end orchestration using managed Airflow, ensuring reliable
  scheduling, dependency management, monitoring, and recovery.
  
• Define
  and track platform SLAs, SLOs, and SLIs, and lead incident triage,
  root cause analysis, and corrective actions.
  
• Implement FinOps best practices for monitoring, optimizing, and allocating
  Databricks and cloud infrastructure costs.
  
• Continuously
  evaluate and improve platform tooling, architecture, reliability,
  security, performance, and developer productivity.
  
• Provide
  technical guidance and mentorship to engineers and establish engineering
  best practices, standards, and reusable patterns.
  
• Partner
  with Data Engineering, Analytics, Architecture, Security, and
  Infrastructure teams to ensure seamless delivery of enterprise data
  solutions.
  
• Contribute
  to cloud infrastructure, cybersecurity, disaster recovery, monitoring,
  logging, and operational readiness.
  
• Drive
  performance optimization and reliability improvements for production data
  workloads.
  

REQUIRED
QUALIFICATIONS

• 10–12
  years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE,
  or related technical disciplines.
  
• Strong
  hands-on experience with Databricks in production environments,
  including workspace and cluster management, Jobs/Workflows, Unity Catalog,
  and integration with orchestration platforms.
  
• Strong
  experience with Microsoft Azure and cloud-native data platforms.
  
• Strong
  knowledge of CI/CD and Git-based development workflows, using tools
  such as Azure DevOps, GitHub Actions, GitLab CI, or similar.
  
• Strong
  experience with Terraform / Infrastructure as Code and cloud
  infrastructure automation.
  
• Hands-on
  experience with Apache Airflow or managed Airflow for data pipeline
  orchestration.
  
• Strong
  programming/scripting skills in Python, Bash, or PowerShell.
  
• Experience
  implementing automated testing, validation, and quality gates for data
  pipelines and data models.
  
• Experience
  with production workload management, including monitoring, logging,
  troubleshooting, performance tuning, and incident management.
  
• Strong
  understanding of cloud security, IAM, RBAC, data access controls, and
  governance.
  
• Experience
  with FinOps, cloud cost optimization, and resource utilization
  monitoring.
  
• Ability
  to provide technical leadership, mentor engineers, and drive engineering
  standards across teams.
  
• Strong
  communication and stakeholder management skills with the ability to work
  effectively with technical and business stakeholders.
  

PREFERRED
QUALIFICATIONS

• Bachelor’s
  degree in Computer Science, Information Technology, Engineering, or a
  related field, or equivalent professional experience.
  
• Experience
  with dbt, SQL, Delta Lake, and modern data engineering practices.
  
• Experience
  with Unity Catalog and enterprise data governance.
  
• Experience
  designing reusable platform frameworks and accelerators.
  
• Experience
  in CPG, retail, manufacturing, or distribution environments.
  
• Experience
  with disaster recovery, business continuity, and highly available cloud
  architectures.
  
• Databricks
  or Azure certifications are a plus.