Industry Consulting Manager

Design and Development: Build and maintain real-time data streaming pipelines using Apache Kafka, including producer and consumer applications. Testing Lifecycle: Write Python scripts for performing unit testing around Kafka topics, consumers, producers, and data processing Lambda functions. AWS Services: Utilise strong knowledge of AWS services, particularly those relevant to stream processing and serverless components like Lambda Functions. Performance Monitoring: Monitor and troubleshoot Kafka cluster performance issues to ensure optimal functionality. Collaboration: Work closely with cross-functional teams to integrate Kafka into various platform components and support existing implementations. Version Control: Manage code using Git, ensuring best practices in version control are followed. Infrastructure as Code (IaC): Apply knowledge of Terraform for infrastructure management where applicable. CI/CD Practices: Implement CI/CD processes using GitHub Actions or similar tools to automate deployment workflows. Strong proficiency in Apache Kafka, AWS MSK, and Confluent Kafka. Excellent programming skills in Python with the ability to write efficient scripts for testing and data processing. Solid understanding of AWS services used in stream processing environments, especially serverless architectures. Familiarity with Git concepts for version control. Knowledge of Infrastructure as Code principles, particularly using Terraform. Experience with CI/CD tools like GitHub Actions is a plus.