Databricks Data Engineer

Job Summary

We are seeking a skilled Databricks Data Engineer to design, build, and optimize scalable data pipelines and analytics solutions on modern cloud platforms. The ideal candidate should have strong experience with Apache Spark, Databricks, and cloud-based data engineering, along with a solid understanding of data warehousing and ETL processes.

Key Responsibilities

• Design and develop scalable data pipelines using Databricks (Delta Lake, Spark, Notebooks)
• Build and maintain ETL/ELT workflows for structured and unstructured data
• Implement data ingestion frameworks from multiple sources (APIs, databases, streaming)
• Optimize Spark jobs for performance, scalability, and cost efficiency
• Work with Azure/AWS/GCP cloud services for data storage and processing
• Develop and manage data models and data lakes/lakehouse architectures
• Ensure data quality, governance, and security across pipelines
• Collaborate with data scientists, analysts, and business stakeholders
• Automate deployments using CI/CD pipelines
• Monitor and troubleshoot production data pipelines

Required Skills & Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field
• 3+ years of experience in data engineering
• Hands-on experience with Databricks
• Strong programming skills in Python / Scala / SQL
• Expertise in Apache Spark (PySpark preferred)
• Experience with Delta Lake, Data Lake, Lakehouse architecture
• Knowledge of Azure Data Factory, AWS Glue, or similar tools
• Strong SQL and database skills (RDBMS, NoSQL)
• Familiarity with data warehousing concepts (Star/Snowflake schema)
• Experience with version control tools like Git

Preferred Skills

• Experience with streaming technologies (Kafka, Structured Streaming)
• Knowledge of Azure Synapse Analytics or Snowflake
• Exposure to DevOps tools (Jenkins, Terraform, Docker)
• Understanding of data governance and security frameworks
• Certification in Databricks or Cloud platforms

Key Competencies

• Strong problem-solving and analytical skills
• Ability to work in Agile/Scrum environments
• Excellent communication and collaboration skills
• Attention to detail and commitment to data quality