DataOps Engineer

Your responsibilities will include: Collaborate with the broader team such as the Data Scientists, Machine Learning Engineers, and Architects to ensure that the data infrastructure supports the solution and enable us as a collective team to produce accurate actionable insights. Design, implement and maintain data infrastructure, ensuring that is scalable, reliable, performant, and efficient. Perform and prepare multiple data pipelines to integrate data, be responsible for data quality control, and formulate data integrity solutions. Implement proactive monitoring, alerting, trend analysis, and robust applications. Implement and comply with the data governance policy and related controls across multiple data platforms. Implement automation tools and practises to streamline data pipeline management and reduce manual effort. Work effectively as part of an Agile team and collaborate well with your other team members. Provide support to other teams if any optimization or the troubleshooting on the performance of the application to avoid errors. Assist in driving out the Data Engineering practices and standards to establish a high confident Data Engineering team. Essential. Experience with DevOps practices, including automated deployment, CI and CD. Relevant cloud certification at professional or associate level would be advantageous. Strong communication and collaboration skills. Agile exposure, Kanban, or Scrum In-depth knowledge of data as a product & Information best practices. Intermediate to advanced level experience in designing, building, and managing data pipelines for batch and streaming applications. Experience in using a wide range for data tools such as AWS services - S3, SFTP, Glue, EMR (Spark), Step Functions, Athena, CloudWatch, CouldTrail, KMS, Kinesis, OpenSearch, etc. Exposure to additional tools such as Hadoop, Spark, Hive, Cassandra, Airflow, Kafka and Flink would be advantageous. Experience with performance tuning of data applications. Experience with performance tuning streaming-based applications for real-time data processing using AWS Kinesis, AWS Kinesys Data Firehose, OpenSearch or similar tools. Working experience within the ML and Analytics lifecycle capabilities such as Data Pipelines, Data Processing, Data Storing, Model Lifecycle, Data Operations, Data Management & Data Governance. Working experience with Cloud platforms such as AWS and GCP. Working experience with Kubernetes and Docker containers. Working experience with CI/CD, IAC and DevOps tools such as CDK, Code Repos, etc. Strong programming skills in Python and SQL.