AI QA Engineer

Immediate joiners required !! 2 days WFO - Hinjewadi Phase 3

Fulcrum Digital is an agile and next -generation
digital accelerating company providing digital transformation and technology
services right from ideation to implementation. These services have
applicability across a variety of industries, including banking & financial
services, insurance, retail, higher education, food, healthcare, and
manufacturing.

• Design and execute test cases for LLM agents, RAG pipelines,
  agentic workflows, and AI -assisted decision tools
  
• Validate AI outputs against ground truth using
  structured accuracy scoring (NAICS, risk exposure flags, hazard group
  mapping)
  
• Detect hallucinations, reasoning gaps, source fabrication,
  and misattributions in model -generated content
  
• Run multi -model comparative testing across GPT, Claude,
  Gemini, and Perplexity — evaluating accuracy, latency, and output
  completeness
  
• Test prompt versions iteratively and track accuracy
  changes across prompt cycles
  
• Validate citation accuracy, document ingestion pipelines,
  and cross -document context handling
  
• Design edge case and negative tests for AI -specific
  failure modes — content filter triggers, tool call limits, missing
  documents, and incomplete synthesis
  
• Perform regression testing after model upgrades, prompt
  changes, and backend fixes, and maintain structured QA sign -off in JIRA
  

What Makes This Role Different from
Traditional QA

• You evaluate whether an AI is reasoning correctly — not
  just whether the UI behaves as expected
  
• You build evaluation rubrics for non -deterministic outputs and apply LLM -as -a -Judge techniques to score quality at scale
  
• You treat every model or prompt change as a potential
  accuracy regression, not just a functional one
  
• You understand that in live AI systems, a passing test today
  does not guarantee a passing test tomorrow
  

Required Skillsets

• 7–8 years of QA experience with minimum 2 years in
  Generative AI / LLM -based projects
  
• Hands -on experience testing chatbots, RAG systems, or
  agentic AI pipelines
  
• Proven ability to perform ground truth validation and
  detect hallucinations and reasoning failures
  
• Familiarity with multi -model evaluation, prompt -aware
  testing, and JIRA -based defect reporting
  

Preferred Skillsets

• Background in insurance or regulated industries;
  exposure to underwriting or risk classification concepts
  
• Familiarity with Azure OpenAI, AWS Bedrock, or
  SharePoint -integrated AI environments
  
• Knowledge of AI governance, content filtering, and PII
  redaction validation