DE-Spark,Scala,kafka

5–10 years Data Engineering

• 
  

  Strong Scala programming

  
  


• 
  

  Strong Apache Spark

  
  


• 
  

  Strong Apache Kafka

  
  


• 
  

  Spark Structured Streaming

  
  


• 
  

  Advanced SQL

  
  


• 
  

  ETL/ELT & Data Pipelines

  
  


• 
  

  Distributed Systems

  
  


• 
  

  Cloud/Databricks exposure

  
  


• Required Technical Skills1. Apache Spark – Mandatory
  

  Strong hands-on experience with:

  
  
  
  
  • Apache Spark
  
  
  • Spark Core
  
  
  • Spark SQL
  
  
  • Spark DataFrame
  
  
  • Spark Dataset
  
  
  • Spark Structured Streaming
  
  
  • Spark transformations and actions
  
  
  • RDD concepts
  
  
  • Joins and aggregations
  
  
  • Window functions
  
  
  • Partitioning and repartitioning
  
  
  • Caching and persistence
  
  
  • Broadcast joins
  
  
  • Handling data skew
  
  
  • Spark job optimization
  
  
  • Spark cluster execution and troubleshooting
  
  
  
  2. Scala – Mandatory
  
  
  
  • Strong programming experience in Scala.
  
  
  • Functional programming concepts.
  
  
  • Collections and higher-order functions.
  
  
  • Case classes, traits, objects, pattern matching.
  
  
  • Exception handling and reusable code development.
  
  
  • Development of Spark applications using Scala.
  
  
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  
  
  
  3. Apache Kafka – Mandatory
  

  Strong practical experience with:

  
  
  
  
  • Kafka architecture
  
  
  • Kafka brokers
  
  
  • Topics and partitions
  
  
  • Producers and consumers
  
  
  • Consumer groups
  
  
  • Offsets and offset management
  
  
  • Replication factor
  
  
  • Partition strategy
  
  
  • Message retention
  
  
  • Kafka Producer/Consumer APIs
  
  
  • Kafka Streams / Kafka Connect
  
  
  • Schema Registry
  
  
  • Avro / JSON serialization
  
  
  • Consumer lag monitoring
  
  
  • Kafka troubleshooting
  
  
  • Kafka performance tuning
  
  
  • Integration of Kafka with Spark Structured Streaming
  
  
  
  4. SQL – Mandatory
  

  Strong SQL skills including:

  
  
  
  
  • Complex SQL queries
  
  
  • Joins
  
  
  • Subqueries
  
  
  • CTEs
  
  
  • Window functions
  
  
  • Aggregations
  
  
  • Query optimization
  
  
  • Data validation and reconciliation
  
  
  • Stored procedures/functions where applicable
  
  
  • Relational database concepts
  
  
  
  5. Data Engineering
  

  Strong understanding of:

  
  
  
  
  • ETL / ELT
  
  
  • Batch processing
  
  
  • Real-time/streaming processing
  
  
  • Data pipelines
  
  
  • Data lakes
  
  
  • Data warehouses
  
  
  • Data modeling
  
  
  • Dimensional modeling
  
  
  • Distributed systems
  
  
  • Data partitioning
  
  
  • Data quality
  
  
  • Data governance
  
  
  • Large-scale data processing
  
  
  
  Cloud / Big Data Experience
  

  Experience with at least one cloud platform is preferred:

  
  


• Required Technical Skills1. Apache Spark – Mandatory
  

  Strong hands-on experience with:

  
  
  
  
  • Apache Spark
  
  
  • Spark Core
  
  
  • Spark SQL
  
  
  • Spark DataFrame
  
  
  • Spark Dataset
  
  
  • Spark Structured Streaming
  
  
  • Spark transformations and actions
  
  
  • RDD concepts
  
  
  • Joins and aggregations
  
  
  • Window functions
  
  
  • Partitioning and repartitioning
  
  
  • Caching and persistence
  
  
  • Broadcast joins
  
  
  • Handling data skew
  
  
  • Spark job optimization
  
  
  • Spark cluster execution and troubleshooting
  
  
  
  2. Scala – Mandatory
  
  
  
  • Strong programming experience in Scala.
  
  
  • Functional programming concepts.
  
  
  • Collections and higher-order functions.
  
  
  • Case classes, traits, objects, pattern matching.
  
  
  • Exception handling and reusable code development.
  
  
  • Development of Spark applications using Scala.
  
  
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  
  
  
  3. Apache Kafka – Mandatory
  

  Strong practical experience with:

  
  
  
  
  • Kafka architecture
  
  
  • Kafka brokers
  
  
  • Topics and partitions
  
  
  • Producers and consumers
  
  
  • Consumer groups
  
  
  • Offsets and offset management
  
  
  • Replication factor
  
  
  • Partition strategy
  
  
  • Message retention
  
  
  • Kafka Producer/Consumer APIs
  
  
  • Kafka Streams / Kafka Connect
  
  
  • Schema Registry
  
  
  • Avro / JSON serialization
  
  
  • Consumer lag monitoring
  
  
  • Kafka troubleshooting
  
  
  • Kafka performance tuning
  
  
  • Integration of Kafka with Spark Structured Streaming
  
  
  
  4. SQL – Mandatory
  

  Strong SQL skills including:

  
  
  
  
  • Complex SQL queries
  
  
  • Joins
  
  
  • Subqueries
  
  
  • CTEs
  
  
  • Window functions
  
  
  • Aggregations
  
  
  • Query optimization
  
  
  • Data validation and reconciliation
  
  
  • Stored procedures/functions where applicable
  
  
  • Relational database concepts
  
  
  
  5. Data Engineering
  

  Strong understanding of:

  
  
  
  
  • ETL / ELT
  
  
  • Batch processing
  
  
  • Real-time/streaming processing
  
  
  • Data pipelines
  
  
  • Data lakes
  
  
  • Data warehouses
  
  
  • Data modeling
  
  
  • Dimensional modeling
  
  
  • Distributed systems
  
  
  • Data partitioning
  
  
  • Data quality
  
  
  • Data governance
  
  
  • Large-scale data processing
  
  
  
  Cloud / Big Data Experience
  

  Experience with at least one cloud platform is preferred:

  
  


• Required Technical Skills1. Apache Spark – Mandatory
  

  Strong hands-on experience with:

  
  
  
  
  • Apache Spark
  
  
  • Spark Core
  
  
  • Spark SQL
  
  
  • Spark DataFrame
  
  
  • Spark Dataset
  
  
  • Spark Structured Streaming
  
  
  • Spark transformations and actions
  
  
  • RDD concepts
  
  
  • Joins and aggregations
  
  
  • Window functions
  
  
  • Partitioning and repartitioning
  
  
  • Caching and persistence
  
  
  • Broadcast joins
  
  
  • Handling data skew
  
  
  • Spark job optimization
  
  
  • Spark cluster execution and troubleshooting
  
  
  
  2. Scala – Mandatory
  
  
  
  • Strong programming experience in Scala.
  
  
  • Functional programming concepts.
  
  
  • Collections and higher-order functions.
  
  
  • Case classes, traits, objects, pattern matching.
  
  
  • Exception handling and reusable code development.
  
  
  • Development of Spark applications using Scala.
  
  
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  
  
  
  3. Apache Kafka – Mandatory
  

  Strong practical experience with:

  
  
  
  
  • Kafka architecture
  
  
  • Kafka brokers
  
  
  • Topics and partitions
  
  
  • Producers and consumers
  
  
  • Consumer groups
  
  
  • Offsets and offset management
  
  
  • Replication factor
  
  
  • Partition strategy
  
  
  • Message retention
  
  
  • Kafka Producer/Consumer APIs
  
  
  • Kafka Streams / Kafka Connect
  
  
  • Schema Registry
  
  
  • Avro / JSON serialization
  
  
  • Consumer lag monitoring
  
  
  • Kafka troubleshooting
  
  
  • Kafka performance tuning
  
  
  • Integration of Kafka with Spark Structured Streaming
  
  
  
  4. SQL – Mandatory
  

  Strong SQL skills including:

  
  
  
  
  • Complex SQL queries
  
  
  • Joins
  
  
  • Subqueries
  
  
  • CTEs
  
  
  • Window functions
  
  
  • Aggregations
  
  
  • Query optimization
  
  
  • Data validation and reconciliation
  
  
  • Stored procedures/functions where applicable
  
  
  • Relational database concepts
  
  
  
  5. Data Engineering
  

  Strong understanding of:

  
  
  
  
  • ETL / ELT
  
  
  • Batch processing
  
  
  • Real-time/streaming processing
  
  
  • Data pipelines
  
  
  • Data lakes
  
  
  • Data warehouses
  
  
  • Data modeling
  
  
  • Dimensional modeling
  
  
  • Distributed systems
  
  
  • Data partitioning
  
  
  • Data quality
  
  
  • Data governance
  
  
  • Large-scale data processing
  
  
  
  Cloud / Big Data Experience


• 
  

  Required Technical Skills1. Apache Spark – Mandatory

  
  

  Strong hands-on experience with:

  
  
  
  
  • Apache Spark
  
  
  • Spark Core
  
  
  • Spark SQL
  
  
  • Spark DataFrame
  
  
  • Spark Dataset
  
  
  • Spark Structured Streaming
  
  
  • Spark transformations and actions
  
  
  • RDD concepts
  
  
  • Joins and aggregations
  
  
  • Window functions
  
  
  • Partitioning and repartitioning
  
  
  • Caching and persistence
  
  
  • Broadcast joins
  
  
  • Handling data skew
  
  
  • Spark job optimization
  
  
  • Spark cluster execution and troubleshooting
  
  
  
  

  2. Scala – Mandatory

  
  
  
  
  • Strong programming experience in Scala.
  
  
  • Functional programming concepts.
  
  
  • Collections and higher-order functions.
  
  
  • Case classes, traits, objects, pattern matching.
  
  
  • Exception handling and reusable code development.
  
  
  • Development of Spark applications using Scala.
  
  
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  
  
  
  

  3. Apache Kafka – Mandatory

  
  

  Strong practical experience with:

  
  
  
  
  • Kafka architecture
  
  
  • Kafka brokers
  
  
  • Topics and partitions
  
  
  • Producers and consumers
  
  
  • Consumer groups
  
  
  • Offsets and offset management
  
  
  • Replication factor
  
  
  • Partition strategy
  
  
  • Message retention
  
  
  • Kafka Producer/Consumer APIs
  
  
  • Kafka Streams / Kafka Connect
  
  
  • Schema Registry
  
  
  • Avro / JSON serialization
  
  
  • Consumer lag monitoring
  
  
  • Kafka troubleshooting
  
  
  • Kafka performance tuning
  
  
  • Integration of Kafka with Spark Structured Streaming
  
  
  
  

  4. SQL – Mandatory

  
  

  Strong SQL skills including:

  
  
  
  
  • Complex SQL queries
  
  
  • Joins
  
  
  • Subqueries
  
  
  • CTEs
  
  
  • Window functions
  
  
  • Aggregations
  
  
  • Query optimization
  
  
  • Data validation and reconciliation
  
  
  • Stored procedures/functions where applicable
  
  
  • Relational database concepts
  
  
  
  

  5. Data Engineering

  
  

  Strong understanding of:

  
  
  
  
  • ETL / ELT
  
  
  • Batch processing
  
  
  • Real-time/streaming processing
  
  
  • Data pipelines
  
  
  • Data lakes
  
  
  • Data warehouses
  
  
  • Data modeling
  
  
  • Dimensional modeling
  
  
  • Distributed systems
  
  
  • Data partitioning
  
  
  • Data quality
  
  
  • Data governance
  
  
  • Large-scale data processing
  
  
  
  

  Cloud / Big Data Experience

  
  

  Experience with at least one cloud platform is preferred:

  
  

  Experience with at least one cloud platform is preferred:

  
  


• Required Technical Skills1. Apache Spark – Mandatory
  

  Strong hands-on experience with:

  
  
  
  
  • Apache Spark
  
  
  • Spark Core
  
  
  • Spark SQL
  
  
  • Spark DataFrame
  
  
  • Spark Dataset
  
  
  • Spark Structured Streaming
  
  
  • Spark transformations and actions
  
  
  • RDD concepts
  
  
  • Joins and aggregations
  
  
  • Window functions
  
  
  • Partitioning and repartitioning
  
  
  • Caching and persistence
  
  
  • Broadcast joins
  
  
  • Handling data skew
  
  
  • Spark job optimization
  
  
  • Spark cluster execution and troubleshooting
  
  
  
  2. Scala – Mandatory
  
  
  
  • Strong programming experience in Scala.
  
  
  • Functional programming concepts.
  
  
  • Collections and higher-order functions.
  
  
  • Case classes, traits, objects, pattern matching.
  
  
  • Exception handling and reusable code development.
  
  
  • Development of Spark applications using Scala.
  
  
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  
  
  
  3. Apache Kafka – Mandatory
  

  Strong practical experience with:

  
  
  
  
  • Kafka architecture
  
  
  • Kafka brokers
  
  
  • Topics and partitions
  
  
  • Producers and consumers
  
  
  • Consumer groups
  
  
  • Offsets and offset management
  
  
  • Replication factor
  
  
  • Partition strategy
  
  
  • Message retention
  
  
  • Kafka Producer/Consumer APIs
  
  
  • Kafka Streams / Kafka Connect
  
  
  • Schema Registry
  
  
  • Avro / JSON serialization
  
  
  • Consumer lag monitoring
  
  
  • Kafka troubleshooting
  
  
  • Kafka performance tuning
  
  
  • Integration of Kafka with Spark Structured Streaming
  
  
  
  4. SQL – Mandatory
  

  Strong SQL skills including:

  
  
  
  
  • Complex SQL queries
  
  
  • Joins
  
  
  • Subqueries
  
  
  • CTEs
  
  
  • Window functions
  
  
  • Aggregations
  
  
  • Query optimization
  
  
  • Data validation and reconciliation
  
  
  • Stored procedures/functions where applicable
  
  
  • Relational database concepts
  
  
  
  5. Data Engineering
  

  Strong understanding of:

  
  
  
  
  • ETL / ELT
  
  
  • Batch processing
  
  
  • Real-time/streaming processing
  
  
  • Data pipelines
  
  
  • Data lakes
  
  
  • Data warehouses
  
  
  • Data modeling
  
  
  • Dimensional modeling
  
  
  • Distributed systems
  
  
  • Data partitioning
  
  
  • Data quality
  
  
  • Data governance
  
  
  • Large-scale data processing
  
  
  
  Cloud / Big Data Experience
  

  Experience with at least one cloud platform is preferred:

  
  


• Required Technical Skills1. Apache Spark – Mandatory
  

  Strong hands-on experience with:

  
  
  
  
  • Apache Spark
  
  
  • Spark Core
  
  
  • Spark SQL
  
  
  • Spark DataFrame
  
  
  • Spark Dataset
  
  
  • Spark Structured Streaming
  
  
  • Spark transformations and actions
  
  
  • RDD concepts
  
  
  • Joins and aggregations
  
  
  • Window functions
  
  
  • Partitioning and repartitioning
  
  
  • Caching and persistence
  
  
  • Broadcast joins
  
  
  • Handling data skew
  
  
  • Spark job optimization
  
  
  • Spark cluster execution and troubleshooting
  
  
  
  2. Scala – Mandatory
  
  
  
  • Strong programming experience in Scala.
  
  
  • Functional programming concepts.
  
  
  • Collections and higher-order functions.
  
  
  • Case classes, traits, objects, pattern matching.
  
  
  • Exception handling and reusable code development.
  
  
  • Development of Spark applications using Scala.
  
  
  • Ability to write clean, modular, scalable, and maintainable Scala code.
  
  
  
  3. Apache Kafka – Mandatory
  

  Strong practical experience with:

  
  
  
  
  • Kafka architecture
  
  
  • Kafka brokers
  
  
  • Topics and partitions
  
  
  • Producers and consumers
  
  
  • Consumer groups
  
  
  • Offsets and offset management
  
  
  • Replication factor
  
  
  • Partition strategy
  
  
  • Message retention
  
  
  • Kafka Producer/Consumer APIs
  
  
  • Kafka Streams / Kafka Connect
  
  
  • Schema Registry
  
  
  • Avro / JSON serialization
  
  
  • Consumer lag monitoring
  
  
  • Kafka troubleshooting
  
  
  • Kafka performance tuning
  
  
  • Integration of Kafka with Spark Structured Streaming
  
  
  
  4. SQL – Mandatory
  

  Strong SQL skills including:

  
  
  
  
  • Complex SQL queries
  
  
  • Joins
  
  
  • Subqueries
  
  
  • CTEs
  
  
  • Window functions
  
  
  • Aggregations
  
  
  • Query optimization
  
  
  • Data validation and reconciliation
  
  
  • Stored procedures/functions where applicable
  
  
  • Relational database concepts
  
  
  
  5. Data Engineering
  

  Strong understanding of:

  
  
  
  
  • ETL / ELT
  
  
  • Batch processing
  
  
  • Real-time/streaming processing
  
  
  • Data pipelines
  
  
  • Data lakes
  
  
  • Data warehouses
  
  
  • Data modeling
  
  
  • Dimensional modeling
  
  
  • Distributed systems
  
  
  • Data partitioning
  
  
  • Data quality
  
  
  • Data governance
  
  
  • Large-scale data processing
  
  
  
  Cloud / Big Data Experience
  

  Experience with at least one cloud platform is preferred: