AI Evaluation Engineer - Part-Time

• Develop and maintain AI evaluation frameworks, regression test suites and automated evaluation pipelines.
• Build datasets, quality dashboards and evaluation tools to improve the performance of AI agents and LLM-powered applications.
• Design evaluation methods such as LLM-as-a-Judge, multi-turn conversations, tool evaluation and agent workflow analysis.
• Integrate AI evaluation into CI/CD and model release processes.
• Analyse production issues and improve datasets, prompts, evaluation logic and AI workflows.
• Conduct experiments to optimise model accuracy, retrieval performance, latency and cost.
• Build reusable platform capabilities and developer tools to support multiple AI products.
• Prepare technical documentation and collaborate with cross-functional engineering teams.

Requirements

• Bachelor's Degree in Computer Science, Software Engineering or a related discipline.
• Minimum 2 years of relevant experience in AI Platform Engineering, LLMOps, Machine Learning Engineering or Software Engineering.
• Experience with most of the following technologies: LangSmith, LangGraph / LangChain, FastAPI, Temporal, Grafana, Redis, React / TypeScript, EvalsHub or similar AI evaluation tools
• Experience with LLM evaluation, Retrieval-Augmented Generation (RAG), semantic search, embeddings or model benchmarking is preferred.
• Strong analytical and problem-solving skills.
• Good communication and stakeholder management skills.
• Able to work independently and collaboratively in a fast-paced environment.

Interested candidates are encouraged to submit their resumes outlining their relevant experience and achievements to sherriyang(@)talentvis.com or click apply!

**We regret to inform that only shortlisted candidates would be notified**

EA License No: 04C3537

EA Personnel No: R22106683

EA Personnel Name: Yang Hui Shan, Sherri