Databricks Solution Architect

• Lead the end-to-end architecture and solution design for enterprise data platforms on Azure, with strong focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native data ecosystems.


• Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers to support reporting, advanced analytics, ML, and business-critical decisioning use cases.


• Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables for batch and near real-time processing.


• Own architecture standards for data modeling, medallion design, reusable engineering patterns, CI/CD, code quality, environment strategy, and release management across Databricks solutions.


• Drive platform governance and security using Unity Catalog, RBAC/ABAC controls, lineage, auditability, and integration with enterprise governance services such as Purview.


• Optimize solution performance by tuning Spark workloads, cluster policies, partitioning strategy, file sizing, caching, and compute cost management for large-scale data processing.


• Collaborate with business stakeholders, product owners, analysts, architects, and downstream consumers to translate functional and non-functional requirements into scalable technical designs.


• Provide technical leadership to engineering teams by reviewing designs, guiding implementation, resolving architectural bottlenecks, and establishing best practices for Databricks-based delivery.


• Evaluate and recommend Databricks capabilities such as Photon, serverless compute, Lakehouse Federation, and streaming patterns to improve scalability, maintainability, and time to value.


• Ensure strong delivery governance through estimation, technical planning, dependency management, risk mitigation, and Agile execution including sprint planning, backlog refinement, and design reviews.

• 10-15 years of experience in data engineering, cloud data platform design, or enterprise data architecture, with at least 5+ years of strong hands-on experience on Databricks and Azure.
• Bachelor’s degree in computer science, Information Technology, Engineering, or a related discipline; master’s degree is preferred.
• Strong expertise in designing modern data architectures, including Lakehouse, medallion architecture, data modeling, data warehousing, and scalable ingestion and transformation frameworks.
• Deep technical proficiency in Databricks, PySpark, Python, SQL, Delta Lake, Databricks Workflows, Auto Loader, and Delta Live Tables.
• Strong experience with Azure cloud services such as Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure DevOps, and integration of Databricks with broader enterprise cloud ecosystems.
• Hands-on experience in defining architecture standards, reusable design patterns, CI/CD strategy, environment management, and delivery best practices for enterprise-scale data platforms.
• Experience with data governance, lineage, and security frameworks including Unity Catalog, role-based access controls, and integration with enterprise governance tools such as Azure Purview.
• Proven ability to optimize large-scale Spark and Databricks workloads, including performance tuning, cluster sizing, workload management, and cost optimization.
• Experience engaging with business stakeholders, product owners, architects, and engineering teams to convert business requirements into scalable solution designs and implementation roadmaps.
• Strong communication, leadership, and problem-solving skills with the ability to mentor teams, review technical designs, and drive architecture decisions across cross-functional programs.
• Preferred: Knowledge of insurance domain data models, regulatory considerations, and analytics use cases relevant to underwriting, claims, pricing, or risk functions.

Bachelor or Master Degree