Confluent Integration Engineer

This role is for one of the Weekday's clients

Salary range: Rs 600000 - Rs 1000000 (ie INR 6 - 10 LPA)

Min Experience: 3+ years

Location: Pune
JobType: full-time

We are looking for a skilled Confluent Integration Engineer with 3–6 years of experience in designing, developing, implementing, and supporting real-time data integration solutions using Apache Kafka and Confluent Platform. The ideal candidate will have strong hands-on expertise in Kafka-based architectures, Confluent connectors, data pipelines, and event-driven integration patterns.

You will work closely with engineering, application, data, and infrastructure teams to build reliable, scalable, and high-performance streaming solutions. The role requires strong troubleshooting skills and the ability to manage Kafka and Confluent integrations across development, testing, and production environments.

Key Responsibilities

• Design, develop, configure, and maintain real-time data integration pipelines using Apache Kafka and Confluent Platform.
• Implement and manage Kafka Connect frameworks and Confluent connectors for integrating databases, applications, APIs, cloud platforms, and enterprise systems.
• Configure and maintain Source and Sink Connectors, including connector properties, transforms, converters, tasks, and worker configurations.
• Develop scalable event-driven integration solutions using Kafka topics, partitions, consumer groups, producers, and consumers.
• Monitor Kafka clusters, Connect workers, connectors, and data pipelines to ensure availability, reliability, and performance.
• Troubleshoot connector failures, message delivery issues, consumer lag, serialization problems, schema-related issues, and pipeline disruptions.
• Work with Confluent Schema Registry to manage Avro, JSON Schema, or Protobuf schemas and maintain data compatibility.
• Optimize Kafka configurations including partitions, replication, retention, consumer settings, and connector task allocation.
• Implement data transformation and routing requirements using Kafka Connect Single Message Transforms (SMTs) where appropriate.
• Collaborate with application and data teams to understand integration requirements and translate them into robust Kafka-based solutions.
• Support deployment, configuration management, upgrades, and production releases for Kafka and Confluent environments.
• Establish monitoring, logging, alerting, and operational processes for Kafka and connector infrastructure.
• Create technical documentation covering architecture, configurations, integration workflows, troubleshooting procedures, and operational standards.

Must-Have Skills

• 3–6 years of hands-on experience in Kafka and enterprise data integration.
• Strong expertise in Apache Kafka concepts, architecture, and administration.
• Hands-on experience with Confluent Platform and Confluent Kafka.
• Strong experience configuring and managing Kafka Connect and Confluent Connectors.
• Experience working with both Source and Sink Connectors.
• Good understanding of Kafka producers, consumers, topics, partitions, replication, offsets, and consumer groups.
• Experience with Confluent Schema Registry and serialization formats such as Avro, JSON Schema, or Protobuf.
• Strong troubleshooting and performance-tuning capabilities for Kafka and connector-based integrations.
• Good understanding of REST APIs and integration patterns.
• Proficiency in scripting or programming using Java, Python, or similar languages.

Good-to-Have Skills

• Experience with connectors for databases such as Oracle, PostgreSQL, MySQL, or SQL Server.
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Experience with Kafka security, including SSL/TLS, SASL, ACLs, and authentication mechanisms.
• Knowledge of Kubernetes and containerized Confluent deployments.
• Familiarity with monitoring tools such as Prometheus, Grafana, Splunk, or ELK.
• Understanding of CI/CD pipelines and DevOps practices.
• Experience with Kafka Streams or ksqlDB is an advantage.

Qualifications

Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related discipline, along with strong analytical, communication, and problem-solving skills.