Azure Databricks - Technical Lead-Data Engg

Area(s) of responsibility

Job Title: Senior Data Engineer / Senior Developer (Azure Databricks)

Location: Noida / Pune

Experience:  9+ years overall, with 5+ years hands‑on in Azure Databricks & Data Engineering

Role Overview - Senior Data Engineer / Senior Developer with strong hands‑on expertise in Azure Databricks, Azure SQL, and Azure Data Integration services. The role requires deep technical ownership, ability to design and build scalable data pipelines, and support modern data architectures while working closely with architects, product owners, and downstream consumers.

Key Responsibilities

Data Engineering & Development

• Design, develop, and maintain end‑to‑end batch and near‑real‑time data pipelines using Azure Databricks (PySpark/SQL).


• Implement scalable data processing using Medallion Architecture (Bronze / Silver / Gold layers).


• Develop and optimize complex T‑SQL queries, stored procedures, and schemas on Azure SQL Database.


• Perform performance tuning, query optimization, and data model refinement across Azure SQL and Databricks.


• Design and monitor complex ETL / ELT pipelines using Azure Data Factory (ADF).


• Build reusable PySpark frameworks for ingestion, transformation, and validation.


• Implement operational logging, monitoring, and data quality checks.


• Analyze existing legacy data architectures and Design and implement mirrored or modernized equivalents on Azure Databricks.

Security, Governance & Compliance

• Experience on secure data access using: Azure Key Vault, Managed Identities, Service Principals and VNET‑integrated services


• Follow enterprise data security, RBAC, and governance standards.


• Ensure compliance with data privacy and audit requirements.

DevOps & Engineering Discipline

• Use Git / Azure DevOps for: Version control, Pull requests & code reviews CI/CD pipelines for data workloads

Required Skills & Qualifications:

• Handson experience on  - Azure SQL Database, Advanced T‑SQL, Schema design, Query & performance tuning , Azure Data Factory (ADF), Complex ETL / ELT pipeline design, Monitoring and troubleshooting, Azure Databricks, PySpark & SQL, Data Modeling, Medallion Architecture (Bronze / Silver / Gold), Dimensional and analytical models, Data Integration

Nice to Have

• Experience analyzing or migrating legacy data platforms (e.g., Rosetta‑style architectures).


• Exposure to Insurance domain data (Claims, Policy, Underwriting, Reinsurance).


• Experience with Delta Lake optimizations and cost management.


• Azure / Databricks certifications.