Python & PySpark Developer

Summary

Responsibilities

• Design, build, and maintain scalable ETL and ELT pipelines to facilitate seamless data movement and transformation.
  
• Develop high-performance data processing applications using Python and PySpark to handle large-scale datasets.
  
• Integrate and manage various AWS data services to support cloud-based data operations and storage requirements.
  
• Create and optimize SQL queries for data extraction, manipulation, and reporting within complex database environments.
  
• Implement and maintain REST APIs to enable efficient data integration between disparate systems and applications.
  
• Utilize cloud technologies to deploy and manage data solutions, ensuring security and compliance with industry standards.
  
• Collaborate with cross-functional teams to define data requirements and deliver data engineering solutions that meet business objectives.
  

• Experience: 6 to 9 years of professional experience in data engineering or related software development roles.
  
• Core Programming: Proficiency in Python and Scala for developing data-intensive applications.
  
• Big Data Technologies: Strong expertise in PySpark for distributed data processing and analytics.
  
• Data Engineering: Demonstrated ability to design and implement ETL/ELT pipelines and data integration strategies.
  
• Cloud & Infrastructure: Hands-on experience with AWS data services and general cloud technologies.
  
• Database Skills: Advanced knowledge of SQL and relational database management systems.
  
• API Development: Competence in building and consuming REST APIs for data connectivity.
  
• Industry Context: Familiarity with the IT Services sector and its specific data handling challenges.