AI Evaluation Engineer

AI Evaluation Engineer

Location: Pune, India

Seniority: Mid-level (3-6 years)

Purpose: Execute comprehensive evaluations of FNZ's AI agents across the six-pillar framework, working as a generalist while developing specialist expertise in 1-2 pillars.

Key Responsibilities:

• Design and conduct evaluations covering Task Performance, Safety, Efficiency, Groundedness, Robustness, and Suitability

• Create "golden sets" of test examples representing expert judgment on desired agent behaviour

• Develop evaluation rubrics and scoring criteria aligned to FNZ Evaluation Framework principles

• Build comprehensive test suites covering happy paths, edge cases, and adversarial inputs

• Evaluate multi-step agentic workflows: planning, tool selection, execution, error handling

• Assess agent groundedness: verify outputs against knowledge bases, detect hallucinations

• Document findings with clear evidence; collaborate with development teams on remediation

• Contribute to building automated evaluation platform and CI/CD integration

Skills and Experience:

• 3-6 years in software testing, QA engineering, AI/ML development, or data science

• Hands-on test automation skills, experience with ML frameworks highly valuable

• Practical experience evaluating LLM applications, RAG systems, or AI agents

• Understanding of prompt engineering, retrieval-augmented generation, and agent architectures

• Analytical mindset to decompose complex agent behaviours and identify failure modes

• Strong documentation and presentation skill

About FNZ

FNZ is committed to opening up wealth so that everyone, everywhere can invest in their future on their terms. We know the foundation to do that already exists in the wealth management industry, but complexity holds firms back. 

We created wealth’s growth platform to help. We provide a global, end-to-end wealth management platform that integrates modern technology with business and investment operations. All in a regulated financial institution. 

We partner with the world’s leading financial institutions, with over US$2.4 trillion in assets on platform (AoP).

Together with our clients, we empower nearly 30 million people across all wealth segments to invest in their future.