Databricks Platform Engineer

Role Overview

Seasoned Data-bricks Senior Platform Engineer who can implement
scalable, end-to-end data solutions on Data-bricks over AWS. This role is
pivotal in building the foundational infrastructure for data products, ensuring
data quality, and enabling machine learning capabilities

 Key
Responsibilities

• Design and implement end-to-end
  ETL pipelines in Databricks, sourcing data from Amazon S3 and Amazon RDS
  and other sources.
  
• Design and deploy Data-bricks
  workspaces and clusters optimized for performance, scalability, and
  cost-efficiency.
  
• Set up data quality frameworks
  and anomaly detection mechanisms to ensure reliability and trust in data
  products.
  
• Enable machine learning
  capabilities within Data-bricks by configuring necessary services,
  libraries, and integrations.
  
• Collaborate with data
  scientists, engineers, and DevOps teams to ensure seamless platform
  operations.
  
• Automate deployment and
  monitoring using CI/CD pipelines, GitHub
  
• Document architecture,
  workflows, and operational procedures for platform governance.
  

Requirements

Core Platform & Cloud

• 5+ years of experience in
  platform engineering roles
  
• Strong expertise in Databricks
  on AWS
  
• Proficient with Amazon S3,
  Amazon RDS, and IAM roles/policies
  
• Experience with Delta Lake,
  Structured Streaming, and Data-bricks Workflows
  

 ETL & Data
Quality

• Hands-on experience building ETL
  pipelines using Spark
  
• Familiarity with data
  validation, profiling, and anomaly detection frameworks
  
• Knowledge of tools like Great Expectations
  or custom validation scripts
  

ML Platform Enablement

• Experience
  setting up ML infrastructure including:
  
• MLflow for experiment tracking
  and model registry
  
• Feature Store for reusable
  feature pipelines
  
• Data-bricks Runtime for ML
  
• Integration with AWS SageMaker
  or custom ML libraries
  
• Understanding of model lifecycle
  management, though model development is out of scope
  

DevOps & Automation

• Proficiency with GitHub, CI/CD
  pipelines, and Terraform or CloudFormation
  

 Preferred
Qualifications

• Experience working with
  healthcare data
  
• Exposure to data mesh or data
  product architecture principles