Activate JobCopilot
This role is for one of our clients
Industry: Technology, Information and Media
Seniority level: Mid-Senior level
Min Experience: 5 years
Location: Remote (India)
JobType: full-time
₹3,00,000 - ₹10,00,000 a year
We are seeking an experienced and versatile Data Engineer to join our growing data engineering team. In this role, you’ll design and build scalable, high-performance data pipelines and infrastructure using modern Azure and big data technologies. You will collaborate closely with cross-functional teams to deliver clean, accessible, and trusted data that powers advanced analytics, AI/ML models, and key business decisions.
Key Responsibilities
Build Scalable Pipelines: Design and implement robust ETL/ELT pipelines to ingest, transform, and store data from diverse structured and unstructured sources (e.g., APIs, Kafka, MongoDB, cloud services).
Develop Azure Data Solutions: Use Azure Data Factory, Databricks, and related tools to manage orchestration, transformation, and pipeline deployment.
Architect & Model Data: Develop scalable data lake and data warehouse solutions using ADLS Gen2, Delta Lake, and Azure SQL DB. Implement optimized data models for analytics and reporting use cases.
Optimize Performance: Apply best practices for data pipeline performance tuning, resource optimization, and parallel processing using Apache Spark and PySpark.
Enable Automation: Automate data workflows and validation processes, including test cases for ETL and Big Data pipelines, ensuring reliability and efficiency.
Collaborate Across Teams: Partner with analysts, data scientists, engineers, and business stakeholders to understand requirements and deliver impactful data products.
Implement Governance & Security: Ensure adherence to data governance, quality, privacy, and compliance standards. Use tools like Key Vault and DevOps CI/CD for secure and automated deployment.
Monitor & Maintain: Establish monitoring, alerting, and logging for data pipeline health and quality, proactively resolving any failures or inconsistencies.
What We’re Looking For
Hands-on Expertise in:
Programming: Python, PySpark, Scala
Azure: Data Factory, Databricks, Key Vault, DevOps CI/CD
Storage: ADLS Gen2, Delta Lake, Azure SQL DB
Big Data Ecosystem: Apache Spark, Hadoop
Experience With:
Data ingestion from real-time and batch sources like Kafka, MongoDB
Building secure, reusable, and scalable data infrastructure
Agile methodology and DevOps principles
Automation testing frameworks for ETL/Big Data pipelines
Preferred (but not required):
Exposure to AI/ML workflows and data science teams
Understanding of MLOps and deployment of ML models at scale
Core Competencies
Strong grasp of data modeling, warehousing, and distributed processing
Solid problem-solving and debugging skills
Passion for clean, reliable, and well-documented data systems
Excellent communication and teamwork abilities
Self-starter attitude with a sense of ownership and accountability
Education & Experience
Bachelor's or Master’s degree in Computer Science, Engineering, or a related field
5–8 years of relevant experience in data engineering roles with cloud and big data tech stacks
Your email job alert is now active!
