Databricks (Delta Lake, PySpark) AI Dev

Position Overview

The Data & AI Engineer role is responsible for building scalable data pipelines, developing AI agents, and delivering end-to-end data and AI applications on the enterprise data platform, with a primary focus on Databricks.

This role bridges data engineering, AI engineering, and data science to deliver production-grade solutions that power analytics, automation, and intelligent applications across Suntory Beverage & Food’s global operations.

As part of the GHQ AAA team, you are expected to contribute to the organisation’s enterprise AI architecture by delivering reusable, scalable, and high-performance data and AI solutions.

Key Responsibilities

Data Pipeline Development

• Design, develop, and maintain scalable data pipelines using Databricks (Delta Lake, PySpark, workflows).
• Build robust ETL/ELT processes to ingest, transform, and serve data from multiple enterprise sources (SAP, external data, APIs).
• Ensure data quality, reliability, and performance optimisation across pipelines.
• Implement data models aligned with analytics and AI use cases (e.g., feature-ready datasets).
• Collaborate with data governance teams to ensure compliance with enterprise data standards.

AI Agent Development

• Design and develop AI agents and GenAI solutions (e.g., knowledge assistants, automation agents) using Databricks and Azure AI capabilities.
• Implement Retrieval-Augmented Generation (RAG), prompt engineering, and orchestration logic for enterprise AI use cases.
• Integrate agents with enterprise data sources, vector databases, and APIs.
• Collaborate with business teams to identify and deliver AI-driven automation and productivity use cases.
• Ensure scalability, performance, and responsible AI practices in agent deployment.

Databricks App Development

• Develop end-to-end data and AI applications using Databricks (e.g., notebooks, dashboards, apps, APIs).
• Build interactive analytics or AI-driven applications for business users (e.g., demand planning tools, decision support apps).
• Expose data and AI services via APIs (e.g., Databricks Model Serving, API integration).
• Work with front-end or BI teams (e.g., Power BI, apps) to integrate backend logic into user-facing solutions.
• Optimise application performance and ensure scalability for enterprise usage.

Data Science Development

• Support development of machine learning models (forecasting, optimisation, classification).
• Perform feature engineering and data preparation for modelling.
• Collaborate with Data Scientists to productionise models within Databricks environment.
• Integrate trained models into pipelines and applications for real-time or batch inference.
• Ensure alignment between modelling outputs and business requirements.

Experiences & Skills Requirements

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Data Science, or related field.
• 1–3 years of experience in data engineering, AI engineering, or related roles.
• Strong hands-on experience with Databricks (Delta Lake, PySpark, workflows, model serving).
• Proficiency in Python and SQL for data processing and application development.
• Experience building data pipelines and ETL processes in large-scale environments.
• Practical experience in AI/GenAI development (LLMs, RAG, prompt engineering).
• Familiarity with API development and integration for data and AI services.
• Understanding of machine learning concepts and model deployment workflows.
• Experience with MLOps / DataOps practices (CI/CD, pipeline automation, monitoring) is preferred.
• Knowledge of Azure ecosystem (ADLS, ADF, Azure AI, API Management) is a plus.
• Strong problem-solving skills and ability to work across cross-functional teams.
• Experience in FMCG, supply chain, or commercial analytics is an advantage.