Databricks Architect - Tarrytown, NY [Hybrid]

Job Details:

Role Summary

We are looking for a strong Databricks Architect to lead enterprise-scale Lakehouse architecture and implementation on AWS. The role involves designing scalable data solutions, modernizing existing data platforms, establishing governance standards, and leading architecture discussions with customers.

Key Responsibilities

• Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.


• Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.


• Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.


• Architect scalable batch, streaming, CDC, and real-time data solutions.


• Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.


• Define governance, security, lineage, and data-quality standards using Unity Catalog.


• Establish best practices for performance, scalability, and cost optimization.


• Lead customer workshops, architecture reviews, and solution assessments.


• Define target-state architecture, migration roadmaps, and reusable patterns.

Required Skills & Qualifications

• 15+ years of overall Data & Analytics experience.


• 4+ years of hands-on Databricks experience as specified in the JD.


• Strong experience with Databricks, Delta Lake, Unity Catalog, DLT, and Auto Loader.


• Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.


• AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.


• Experience with Airflow, CDC, and enterprise data migration.


• Knowledge of data governance, security, and observability.


• Strong customer-facing, solution architecture, and technical leadership skills.

Preferred Qualifications

• Healthcare or pharmaceutical domain experience is preferred but not mandatory.


• Experience with enterprise-scale data platform modernization and migration is beneficial.