Python Lead

This role is for one of the Weekday's clients

Salary range: Rs 1000000 - Rs 2000000 (ie INR 10-20 LPA)

Min Experience: 5 years

Location: Pune

JobType: full-time

We are looking for an experienced Python Lead with deep expertise in web scraping and data extraction to lead the design, development, and optimization of scalable scraping solutions. This role requires strong hands-on skills with Scrapy, Beautiful Soup (BeautifulSoup), and web scraping frameworks, along with the ability to guide a team, define best practices, and deliver reliable data pipelines at scale.

As a Python Lead, you will take ownership of complex scraping architectures, ensure high data quality, handle anti-bot challenges, and mentor developers while working closely with product, data, and business stakeholders.

Key Responsibilities

• Lead the end-to-end design and development of web and data scraping solutions using Python.
• Architect and maintain scalable scraping systems using Scrapy framework, Beautiful Soup / BeautifulSoup, and custom Python solutions.
• Handle complex scraping scenarios including dynamic websites, pagination, authentication flows, JavaScript-rendered content, and APIs.
• Implement techniques to manage rate limiting, IP rotation, proxy management, CAPTCHA handling, and bot detection mechanisms.
• Ensure data accuracy, consistency, and reliability, including validation, cleaning, and structured storage of scraped data.
• Optimize scraping performance for speed, scalability, and fault tolerance.
• Lead code reviews, enforce coding standards and best practices, and ensure maintainable and testable codebases.
• Mentor and guide junior developers, providing technical leadership and knowledge sharing.
• Collaborate with cross-functional teams to understand data requirements and translate them into robust scraping solutions.
• Ensure compliance with legal, ethical, and website terms of service related to web scraping activities.

Required Skills & Qualifications

• 5–10 years of hands-on Python development experience, with a strong focus on web scraping and data scraping.
• Extensive experience with Scrapy, Scrapy Framework, and Beautiful Soup / BeautifulSoup.
• Strong understanding of HTML, CSS, XPath, and regular expressions.
• Experience scraping large-scale and high-volume websites.
• Knowledge of handling dynamic content using headless browsers or JavaScript rendering techniques.
• Solid experience with databases (SQL/NoSQL) and data formats such as JSON, CSV, and XML.
• Strong debugging, problem-solving, and performance optimization skills.
• Experience leading or mentoring teams in a technical capacity.

Nice to Have

• Experience with cloud platforms, distributed systems, or task queues.
• Familiarity with data pipelines, ETL processes, or analytics workflows.
• Exposure to monitoring, logging, and alerting for scraping systems.