Senior Pyspark Developer

Company : Kavi Software Technologies

Job Type : Full Time

Chennai, Tamil Nadu

Number of Applicants

:

000+

Apply Now

Let AI Supercharge Your Job Hunt!

JobCopilot scans 500,000+ company career sites daily to find jobs for you

Never miss an opportunity Save hours by auto-filling applications forms Land more interviews with tailored applications

Activate JobCopilot

Job Description - Senior Pyspark Developer

Senior PySpark Developer

Job Summary

We are looking for a Senior PySpark Developer with strong hands -on experience working on the Cloudera stack. The role involves building and maintaining large -scale data processing and ETL pipelines using PySpark. Experience in finance or banking environments is a plus.

Experience: 5+ years

Key Responsibilities

• Develop and maintain PySpark -based ETL pipelines on the Cloudera platform

• Write efficient PySpark and Spark SQL transformations

• Process large volumes of structured and semi -structured data

• Optimize Spark jobs for performance and scalability

• Create and manage Hive tables and partitions

• Handle data validation, reconciliation, and error handling

• Support production deployments and troubleshoot issues

• Work closely with data engineers and business teams

Primary Skill (Mandatory)

• Strong PySpark experience (5+ years)

o DataFrame API

o Spark SQL

o Performance tuning (joins, partitions, shuffles)

o Batch data processing

Required Skills

• Strong Python programming skills

• Good SQL knowledge

• Experience with large -scale data processing

• Git or similar version control tools

Preferred / Nice -to -Have Skills (Not Mandatory)

• Kafka for streaming or data ingestion

• Starburst for distributed SQL querying

• Oracle database integration or data extraction

• Workflow tools such as Airflow or Oozie

• Experience working with the Cloudera stack (HDFS, Hive, YARN, Spark)

Domain Experience (Plus)

• Finance or Banking domain experience

• Exposure to transactional, risk, or regulatory data

Education

• Bachelor’s degree in Computer Science or related field (or equivalent professional experience)

Email address

Why are you reporting this job?

I think it’s a discriminatory or offensive

I think it’s fraudulent or a scam

I think it’s trying to sell something unrelated to the job / it’s asking for money

I think it contains incorrect or broken information

Other

All Job Ads are subject to GrabJobs’s Terms of Service. We allow users to flag postings that may be in violation of those terms. Job Ads may also be flagged by GrabJobs moderation team. However, no moderation system is perfect, and flagging a posting does not ensure that it will be removed.