Data Engineer (Spark/Scala)
About the Role
We are seeking an experienced Data Engineer to design, build, and optimize complex data workflows across on-premises and cloud environments. This role requires deep hands-on expertise in Apache Spark, Databricks, and Scala/PySpark, along with strong SQL and Python skills, to build robust, high-performance data pipelines. You will work extensively on complex on-prem workflows, integrating data across multiple file systems and formats, migrating and modernizing legacy processes, and ensuring efficient, reliable data movement across heterogeneous environments.
Key Responsibilities
Design, develop, and maintain large-scale data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark
Build and support complex on-premises data workflows, including migration/hybrid on-prem-to-cloud integration patterns
Integrate data across diverse file systems (on-prem file shares, NAS, HDFS, S3) and formats JSON, Parquet, Fixed-Length, CSV, Excel, Avro
Write efficient, optimized SQL for data extraction, transformation, and loading across relational databases
Connect to and extract data efficiently from various source databases, tuning queries and pipelines for performance at scale
Develop and maintain workflow orchestration using Airflow (or similar schedulers) for reliable, monitored pipeline execution
Write clean, production-grade Python code for data processing, automation, and tooling
Build and maintain unit/integration tests for data pipelines to ensure data quality and reliability
Create and maintain clear technical documentation for pipelines, data flows, and system architecture
Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues in complex, multi-system workflows
Collaborate with cross-functional teams (data science, analytics, application engineering) to support downstream data consumption
Support cloud integration efforts, particularly with Azure, as workloads evolve from on-prem to hybrid/cloud architectures
Required Qualifications
Primary Skills:
Strong hands-on experience with Apache Spark and Databricks for large-scale data processing
Proficiency with Amazon S3 for data storage and pipeline integration
Strong SQL skills - query optimization, complex joins, performance tuning
Proven experience integrating data across various file systems and formats: JSON, Parquet, Fixed-Length, CSV, Excel, Avro, etc.
Strong knowledge of Scala Spark and PySpark for distributed data processing
Strong Python programming skills for scripting, automation, and data engineering tasks
Strong experience connecting to and efficiently extracting data from databases (relational/other), including performance-conscious extraction strategies
Demonstrated experience working on complex on-prem data workflows (multi-system integration, legacy system data extraction, hybrid on-prem/cloud pipelines)
Experience leveraging coding assistant tools and implementing AI agents to enhance development productivity and task execution.
Secondary Skills:
Experience with Azure cloud services (storage, compute, data services)
Experience with Apache Airflow for workflow orchestration and scheduling
Experience writing automated tests for data pipelines (unit, integration, data quality checks)
Strong documentation skills able to clearly document pipelines, data lineage, and technical designs
Good to Have:
Working knowledge of Java
Familiarity with React for building internal tooling/dashboards
Experience with Prefect for workflow orchestration
PBM (Pharmacy Benefit Management) / Healthcare domain knowledge
Data Engineer (Spark/Scala)
About the Role
We are seeking an experienced Data Engineer to design, build, and optimize complex data workflows across on-premises and cloud environments. This role requires deep hands-on expertise in Apache Spark, Databricks, and Scala/PySpark, along with strong SQL and Python skills, to build robust, high-performance data pipelines. You will work extensively on complex on-prem workflows, integrating data across multiple file systems and formats, migrating and modernizing legacy processes, and ensuring efficient, reliable data movement across heterogeneous environments.
Key Responsibilities
Design, develop, and maintain large-scale data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark
Build and support complex on-premises data workflows, including migration/hybrid on-prem-to-cloud integration patterns
Integrate data across diverse file systems (on-prem file shares, NAS, HDFS, S3) and formats JSON, Parquet, Fixed-Length, CSV, Excel, Avro
Write efficient, optimized SQL for data extraction, transformation, and loading across relational databases
Connect to and extract data efficiently from various source databases, tuning queries and pipelines for performance at scale
Develop and maintain workflow orchestration using Airflow (or similar schedulers) for reliable, monitored pipeline execution
Write clean, production-grade Python code for data processing, automation, and tooling
Build and maintain unit/integration tests for data pipelines to ensure data quality and reliability
Create and maintain clear technical documentation for pipelines, data flows, and system architecture
Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues in complex, multi-system workflows
Collaborate with cross-functional teams (data science, analytics, application engineering) to support downstream data consumption
Support cloud integration efforts, particularly with Azure, as workloads evolve from on-prem to hybrid/cloud architectures
Required Qualifications
Primary Skills:
Strong hands-on experience with Apache Spark and Databricks for large-scale data processing
Proficiency with Amazon S3 for data storage and pipeline integration
Strong SQL skills - query optimization, complex joins, performance tuning
Proven experience integrating data across various file systems and formats: JSON, Parquet, Fixed-Length, CSV, Excel, Avro, etc.
Strong knowledge of Scala Spark and PySpark for distributed data processing
Strong Python programming skills for scripting, automation, and data engineering tasks
Strong experience connecting to and efficiently extracting data from databases (relational/other), including performance-conscious extraction strategies
Demonstrated experience working on complex on-prem data workflows (multi-system integration, legacy system data extraction, hybrid on-prem/cloud pipelines)
Experience leveraging coding assistant tools and implementing AI agents to enhance development productivity and task execution.
Secondary Skills:
Experience with Azure cloud services (storage, compute, data services)
Experience with Apache Airflow for workflow orchestration and scheduling
Experience writing automated tests for data pipelines (unit, integration, data quality checks)
Strong documentation skills able to clearly document pipelines, data lineage, and technical designs
Good to Have:
Working knowledge of Java
Familiarity with React for building internal tooling/dashboards
Experience with Prefect for workflow orchestration
PBM (Pharmacy Benefit Management) / Healthcare domain knowledge.
Data Engineer (Spark/Scala)
About the Role
We are seeking an experienced Data Engineer to design, build, and optimize complex data workflows across on-premises and cloud environments. This role requires deep hands-on expertise in Apache Spark, Databricks, and Scala/PySpark, along with strong SQL and Python skills, to build robust, high-performance data pipelines. You will work extensively on complex on-prem workflows, integrating data across multiple file systems and formats, migrating and modernizing legacy processes, and ensuring efficient, reliable data movement across heterogeneous environments.
Key Responsibilities
Design, develop, and maintain large-scale data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark
Build and support complex on-premises data workflows, including migration/hybrid on-prem-to-cloud integration patterns
Integrate data across diverse file systems (on-prem file shares, NAS, HDFS, S3) and formats JSON, Parquet, Fixed-Length, CSV, Excel, Avro
Write efficient, optimized SQL for data extraction, transformation, and loading across relational databases
Connect to and extract data efficiently from various source databases, tuning queries and pipelines for performance at scale
Develop and maintain workflow orchestration using Airflow (or similar schedulers) for reliable, monitored pipeline execution
Write clean, production-grade Python code for data processing, automation, and tooling
Build and maintain unit/integration tests for data pipelines to ensure data quality and reliability
Create and maintain clear technical documentation for pipelines, data flows, and system architecture
Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues in complex, multi-system workflows
Collaborate with cross-functional teams (data science, analytics, application engineering) to support downstream data consumption
Support cloud integration efforts, particularly with Azure, as workloads evolve from on-prem to hybrid/cloud architectures
Required Qualifications
Primary Skills:
Strong hands-on experience with Apache Spark and Databricks for large-scale data processing
Proficiency with Amazon S3 for data storage and pipeline integration
Strong SQL skills - query optimization, complex joins, performance tuning
Proven experience integrating data across various file systems and formats: JSON, Parquet, Fixed-Length, CSV, Excel, Avro, etc.
Strong knowledge of Scala Spark and PySpark for distributed data processing
Strong Python programming skills for scripting, automation, and data engineering tasks
Strong experience connecting to and efficiently extracting data from databases (relational/other), including performance-conscious extraction strategies
Demonstrated experience working on complex on-prem data workflows (multi-system integration, legacy system data extraction, hybrid on-prem/cloud pipelines)
Experience leveraging coding assistant tools and implementing AI agents to enhance development productivity and task execution.
Secondary Skills:
Experience with Azure cloud services (storage, compute, data services)
Experience with Apache Airflow for workflow orchestration and scheduling
Experience writing automated tests for data pipelines (unit, integration, data quality checks)
Strong documentation skills able to clearly document pipelines, data lineage, and technical designs
Good to Have:
Working knowledge of Java
Familiarity with React for building internal tooling/dashboards
Experience with Prefect for workflow orchestration
PBM (Pharmacy Benefit Management) / Healthcare domain knowledge
Data Engineer (Spark/Scala)
About the Role
We are seeking an experienced Data Engineer to design, build, and optimize complex data workflows across on-premises and cloud environments. This role requires deep hands-on expertise in Apache Spark, Databricks, and Scala/PySpark, along with strong SQL and Python skills, to build robust, high-performance data pipelines. You will work extensively on complex on-prem workflows, integrating data across multiple file systems and formats, migrating and modernizing legacy processes, and ensuring efficient, reliable data movement across heterogeneous environments.
Key Responsibilities
Design, develop, and maintain large-scale data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark
Build and support complex on-premises data workflows, including migration/hybrid on-prem-to-cloud integration patterns
Integrate data across diverse file systems (on-prem file shares, NAS, HDFS, S3) and formats JSON, Parquet, Fixed-Length, CSV, Excel, Avro
Write efficient, optimized SQL for data extraction, transformation, and loading across relational databases
Connect to and extract data efficiently from various source databases, tuning queries and pipelines for performance at scale
Develop and maintain workflow orchestration using Airflow (or similar schedulers) for reliable, monitored pipeline execution
Write clean, production-grade Python code for data processing, automation, and tooling
Build and maintain unit/integration tests for data pipelines to ensure data quality and reliability
Create and maintain clear technical documentation for pipelines, data flows, and system architecture
Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues in complex, multi-system workflows
Collaborate with cross-functional teams (data science, analytics, application engineering) to support downstream data consumption
Support cloud integration efforts, particularly with Azure, as workloads evolve from on-prem to hybrid/cloud architectures
Required Qualifications
Primary Skills:
Strong hands-on experience with Apache Spark and Databricks for large-scale data processing
Proficiency with Amazon S3 for data storage and pipeline integration
Strong SQL skills - query optimization, complex joins, performance tuning
Proven experience integrating data across various file systems and formats: JSON, Parquet, Fixed-Length, CSV, Excel, Avro, etc.
Strong knowledge of Scala Spark and PySpark for distributed data processing
Strong Python programming skills for scripting, automation, and data engineering tasks
Strong experience connecting to and efficiently extracting data from databases (relational/other), including performance-conscious extraction strategies
Demonstrated experience working on complex on-prem data workflows (multi-system integration, legacy system data extraction, hybrid on-prem/cloud pipelines)
Experience leveraging coding assistant tools and implementing AI agents to enhance development productivity and task execution.
Secondary Skills:
Experience with Azure cloud services (storage, compute, data services)
Experience with Apache Airflow for workflow orchestration and scheduling
Experience writing automated tests for data pipelines (unit, integration, data quality checks)
Strong documentation skills able to clearly document pipelines, data lineage, and technical designs
Good to Have:
Working knowledge of Java
Familiarity with React for building internal tooling/dashboards
Experience with Prefect for workflow orchestration
PBM (Pharmacy Benefit Management) / Healthcare domain knowledge
GrabJobs is the no1 job portal in India, connecting you to thousands of jobs fast! Find the best jobs in India, apply in 1 click and get a job today!
Copyright © 2026 Grabjobs Pte.Ltd. All Rights Reserved.
Your email job alert is now active!