Senior / Lead Data Engineer – Service Layer / QTC Integration Platform

Role Details

• Role: Senior / Lead Data Engineer
  
• Primary Skills: Go / Golang, Python, AWS, Kafka / MSK, Microservices, Distributed Systems, Data Engineering
  
• Platform Exposure: AWS Serverless, Kafka, Snowflake, PostgreSQL, ClickHouse, Event-Driven Architecture
  
• Domain Exposure: Quote-to-Cash, CRM, ERP, Customer Lifecycle, Enterprise Data preferred
  
• Experience: 8+ years
  
• Location/ Mode: Bengaluru/Hybrid
  

Role Overview

We are looking for a strong, hands-on, and highly autonomous Senior / Lead Data Engineer to design, build, and operate the Service Layer platform powering enterprise Quote-to-Cash (QTC), customer lifecycle, finance, and enterprise application integrations.

This role requires deep expertise in distributed systems, event-driven architecture, microservices, data engineering, cloud-native development, and platform reliability. The candidate will also provide technical leadership to a team of engineers, driving solution design, engineering standards, delivery execution, and operational support across multiple integration initiatives.

The platform is built using AWS, Kafka / AWS MSK, Go, Python, canonical data models, asynchronous messaging, and cloud-native architecture patterns, processing high-volume, business-critical events with strong focus on reliability, security, scalability, observability, and operational readiness.

Key Responsibilities

Technical Delivery & Engineering Leadership

• Own end-to-end technical delivery of key initiatives, ensuring solutions are delivered with quality, scalability, security, and operational readiness.
  
• Lead and mentor a team of engineers through task planning, technical guidance, code reviews, design reviews, and delivery management.
  
• Drive engineering best practices across software development, CI/CD, observability, resiliency, security, and cloud-native architecture.
  
• Manage multiple concurrent initiatives while balancing business priorities, technical debt, and platform evolution.
  

Architecture & Platform Design

• Drive architecture and design decisions for event-driven integrations, canonical data models, orchestration frameworks, and platform capabilities.
  
• Define and enforce engineering standards, secure-by-design principles, cloud best practices, data governance, and non-functional requirements.
  
• Design and build scalable Service Layer foundation capabilities including messaging, orchestration, observability, reconciliation, data migration, operational tooling, and canonical data management.
  
• Ensure platform design supports resilience, scalability, fault tolerance, performance, and supportability.
  

Event-Driven Integration & Microservices

• Design, develop, and operate real-time event-driven integration services using Go / Golang and Python.
  
• Build and enhance the Service Layer platform to orchestrate Quote-to-Cash workflows across enterprise applications.
  
• Design and implement scalable Kafka / AWS MSK-based event processing with support for:
  • Retries
    
  • Idempotency
    
  • Sequencing
    
  • Message replay
    
  • Fault recovery
    
  • Dead-letter queues
    
  • Eventual consistency
    
• Build secure and scalable RESTful APIs, integration services, and enterprise system interfaces.
  

Data Engineering, Migration & Reconciliation

• Design and implement large-scale ETL, migration, reconciliation, and data-quality solutions using:
  • AWS Glue
    
  • Step Functions
    
  • Snowflake
    
  • PostgreSQL
    
  • ClickHouse
    
• Build and enforce canonical data models, data contracts, transformation layers, and validation frameworks.
  
• Support enterprise integrations by standardizing communication across multiple business systems.
  
• Ensure data accuracy, traceability, consistency, and operational reliability across the platform.
  

AWS Cloud-Native Engineering

• Design and build cloud-native applications on AWS using:
  • AWS Lambda
    
  • API Gateway
    
  • S3
    
  • RDS / Aurora PostgreSQL
    
  • DynamoDB
    
  • SQS
    
  • SNS
    
  • EventBridge
    
  • IAM
    
  • CloudWatch
    
  • VPC
    
• Build high-availability, business-critical applications with strong focus on performance, security, and operational supportability.
  

Observability, Monitoring & Operations

• Design and implement observability, monitoring, alerting, and operational dashboards for end-to-end workflow visibility.
  
• Work with tools such as:
  • CloudWatch
    
  • Grafana
    
  • Datadog
    
  • Prometheus
    
  • OpenTelemetry
    
• Support production operations including incident management, root-cause analysis, troubleshooting, and performance optimization.
  
• Drive metrics-driven improvements for reliability, scalability, and platform performance.
  

Collaboration & Stakeholder Management

• Partner with Product Managers, Architects, Enterprise Data teams, Engineering, DevOps, and business stakeholders.
  
• Define technical roadmaps, execution plans, dependencies, and delivery milestones.
  
• Communicate technical risks, trade-offs, design decisions, and mitigation plans clearly to stakeholders.
  
• Collaborate across multiple teams to ensure integration initiatives are delivered efficiently.
  

Must-Have Skills

Leadership & Delivery

• Proven experience leading and mentoring engineering teams.
  
• Strong experience in task planning, technical guidance, code reviews, and delivery management.
  
• Ability to drive architecture decisions for large-scale distributed systems and enterprise integration platforms.
  
• Experience managing multiple parallel technical initiatives.
  

Programming & Backend Engineering

• Strong hands-on experience in:
  • Go / Golang
    
  • Python
    
• Experience building scalable backend services, event-driven integrations, orchestration workflows, and data processing solutions.
  
• Strong understanding of REST API development and secure API design.
  

AWS Cloud & Serverless

• Strong experience designing and building cloud-native applications on AWS.
  
• Hands-on experience with:
  • Lambda
    
  • API Gateway
    
  • S3
    
  • RDS / Aurora PostgreSQL
    
  • DynamoDB
    
  • SQS / SNS
    
  • EventBridge
    
  • IAM
    
  • CloudWatch
    
  • VPC
    
• Strong understanding of AWS serverless architecture patterns.
  

Kafka / Event-Driven Systems

• Strong experience designing and operating event-driven and asynchronous systems using:
  • Kafka
    
  • AWS MSK
    
  • RabbitMQ or equivalent messaging technologies
    
• Strong understanding of:
  • Fault tolerance
    
  • Idempotency
    
  • Retry patterns
    
  • Dead-letter queues
    
  • Message replay
    
  • Eventual consistency
    
  • Sequencing
    
  • Event orchestration
    

Databases & Data Engineering

• Strong experience with:
  • PostgreSQL
    
  • ClickHouse
    
  • Relational databases
    
  • DynamoDB / NoSQL databases
    
• Strong knowledge of:
  • Data modelling
    
  • Query optimization
    
  • Performance tuning
    
  • Data validation
    
  • Reconciliation frameworks
    
• Experience with AWS Glue, Step Functions, Snowflake, and large-scale data workflows.
  

CI/CD, Security & Observability

• Experience with CI/CD pipelines, automated testing, code quality, and secure delivery using tools such as:
  • GitLab CI
    
  • Jenkins
    
  • SonarQube
    
  • Trivy
    
  • Semgrep
    
• Strong understanding of cloud security and secure-by-design principles including:
  • IAM
    
  • OAuth2
    
  • JWT authentication
    
  • Encryption at rest and in transit
    
  • Secrets Manager
    
  • Vulnerability management
    
  • Audit logging
    
  • Compliance controls
    

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or related discipline.
  
• 8+ years of software engineering experience.
  
• At least 4+ years focused on cloud-native, data engineering, integration, or distributed platforms.
  
• 4+ years building cloud-native solutions on AWS.
  
• 4+ years working with Go / Golang, Python, AWS services, and Kafka in production environments.
  
• Proven experience designing, building, and operating highly available enterprise integration and event-driven platforms.
  
• Strong experience in production support, incident management, root-cause analysis, and performance optimization.
  

Nice to Have

• AWS certifications such as:
  • AWS Certified Data Engineer – Associate
    
  • AWS Certified Solutions Architect – Associate
    
  • AWS Certified Developer – Associate
    
  • Equivalent AWS certifications
    
• Experience in:
  • Quote-to-Cash
    
  • CRM
    
  • ERP
    
  • Customer Lifecycle Management
    
  • Enterprise Data domains
    
• Exposure to large-scale finance, billing, or enterprise integration transformation programmes.
  
• Experience building operational dashboards and platform-level reporting for engineering and business teams.