Databricks SME

Key Responsibilities:

Architecture & Platform Design

• Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework


• Define reference architectures for batch, streaming, analytics, and ML workloads


• Select and standardize cluster, compute, and workspace architectures


• Design multi-workspace strategies (dev/test/prod, shared vs. isolated)


• Ensure architectures meet scalability, availability, and performance requirements

Well-Architected Framework Alignment

Apply Databricks best practices across all pillars, including:

• Security & Governance (Unity Catalog, IAM, data access controls)


• Reliability & Resilience (job retries, checkpointing, failure isolation)


• Performance Efficiency (cluster sizing, autoscaling, caching)


• Cost Optimization (compute policies, workload separation, monitoring)


• Operational Excellence (monitoring, automation, CI/CD, runbooks)

Implementation & Engineering

• Lead Databricks workspace, cluster, and Unity Catalog implementations


• Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming


• Build and optimize ETL/ELT pipelines using Spark and SQL


• Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)


• Establish CI/CD pipelines for notebooks, jobs, and infrastructure

Security, Governance & Compliance

• Implement Unity Catalog for centralized governance


• Define data classification, lineage, and audit strategies


• Enforce least-privilege access and secure networking patterns


• Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)

Operations & Optimization

• Monitor platform health, performance, and cost


• Troubleshoot production issues across jobs, clusters, and data pipelines


• Perform workload tuning and cost-performance optimization


• Define SLOs, alerts, and operational metrics

Collaboration & Leadership

• Partner with Data Engineering, Analytics, ML, Platform, and Security teams


• Translate business requirements into technical architectures


• Provide architectural guidance and technical mentorship


• Communicate risks, tradeoffs, and recommendations to leadership

Required Qualifications:

Experience

• 7+ years in data engineering, analytics, or platform architecture


• 3–5+ years hands-on Databricks experience in production environments


• Proven experience applying the Databricks Well-Architected Framework


• Experience designing cloud-native lakehouse architectures


• Experience supporting mission-critical data platforms

Technical Skills

• Databricks Lakehouse Platform


• Apache Spark (PySpark / Scala / Spark SQL)


• Delta Lake, Delta Live Tables, Structured Streaming


• Unity Catalog (governance, lineage, access controls)


• Cloud platforms: AWS, Azure, or GCP


• Infrastructure as Code (Terraform strongly preferred)


• CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)


• Data formats and protocols (Parquet, JSON, Avro)

Certifications Required:

• Databricks Certified Data Engineer Professional


• Databricks Certified Professional Architect (or equivalent advanced certification) 

Preferred / Additional Certifications

• AWS Certified Solutions Architect (Associate or Professional)


• Azure Solutions Architect Expert


• Google Professional Data Engineer


• Databricks Machine Learning Professional


• Snowflake or other cloud data platform certifications

Soft Skills

• Strong architectural decision-making and documentation skills


• Excellent communication with technical and non-technical stakeholders


• Ability to lead design reviews and architecture governance forums


• Strong troubleshooting and performance-tuning mindset

Nice-to-Have Experience

• MLflow and MLOps architectures


• Real-time analytics and streaming pipelines


• Multi-region or cross-account data architectures


• Consulting or MSP delivery experience