Databricks

⃣ Sr. Data Engineer – Databricks Pipeline Developer

Role Overview

Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.

Key Responsibilities

• Develop PySpark pipelines for
  • API ingestion
  • FTP file ingestion
  • Data cleansing
  • Enrichment transformations
• Implement incremental & batch loads
• Develop Bronze → Silver → Gold transformations
• Translate legacy SQL stored procedures into Spark jobs
• Optimize Delta Lake tables
• Implement validation checks
• Support testing & reconciliation

Required Skills

• Strong PySpark / Spark SQL
• Strong SQL (read & interpret legacy procedures)
• Experience with API integration
• Experience handling flat files (FTP ingestion)
• Experience with Delta Lake
• Knowledge of partitioning & optimization