PySpark Developer

This role is for one of Weekday’s clients
Salary range: Rs 2000000 - Rs 3000000 (ie INR 20 - 30 LPA)

Min Experience: 8+ years
Location: Bengaluru, Chennai, Tamil Nadu, India
JobType: full-time

We are looking for an experienced PySpark Developer with 8–13 years of experience in data engineering, Python programming, and large-scale data processing. The ideal candidate will have strong expertise in developing, optimizing, and maintaining scalable data pipelines using PySpark and Apache Spark. This role requires hands-on experience in processing structured and semi-structured data, implementing ETL/ELT workflows, and building reliable data solutions that support business intelligence, analytics, and data-driven decision-making.

The candidate will collaborate with data engineers, architects, analysts, and other technical stakeholders to design efficient data processing solutions, improve pipeline performance, and ensure data quality across the organization.

Key Responsibilities

1. Data Pipeline Development

• Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
• Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
• Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
• Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
• Integrate data from databases, APIs, cloud storage, and enterprise data platforms.

2. PySpark Development and Optimization

• Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
• Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
• Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
• Work with large datasets while ensuring efficient memory management and distributed computing performance.
• Apply best practices for handling joins, aggregations, data skew, and incremental data processing.

3. Python Programming

• Write clean, modular, reusable, and well-documented Python code for data engineering applications.
• Develop utilities, validation scripts, and automation solutions to streamline data workflows.
• Implement exception handling, logging, testing, and debugging practices to improve code quality.
• Collaborate with engineering teams to maintain coding standards and version control practices.

4. Data Quality and Reliability

• Implement data validation, reconciliation, and monitoring mechanisms across pipelines.
• Identify and resolve data inconsistencies, duplication, and transformation errors.
• Maintain technical documentation, data lineage, and operational procedures.
• Support production deployments, incident resolution, and continuous improvement initiatives.

5. Collaboration and Technical Leadership

• Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions.
• Participate in architecture discussions, code reviews, technical design sessions, and development planning.
• Mentor junior developers and contribute to engineering best practices.

Required Skills and Qualifications

• 8–13 years of overall experience in software development, data engineering, or related technical roles.
• Strong hands-on expertise in PySpark, Apache Spark, and Python.
• Proven experience designing and maintaining scalable data pipelines and ETL/ELT processes.
• Strong understanding of distributed computing, data partitioning, transformations, and performance optimization.
• Experience working with SQL, relational databases, and large-scale structured or semi-structured datasets.
• Familiarity with data warehousing concepts, data modeling, and data quality practices.
• Experience with Git, debugging, testing, and production support.

Preferred Qualifications

• Experience with cloud data platforms such as AWS, Microsoft Azure, or Google Cloud.
• Exposure to platforms such as Databricks, Delta Lake, Snowflake, or cloud-based data lakes.
• Familiarity with orchestration tools such as Apache Airflow and CI/CD practices.
• Knowledge of Kafka or other streaming technologies is an advantage.