Testing Lead -QA

Job Title: Testing Lead – Deep Learning / LLM / VLM

Location

Remote /
Hybrid / On -site

Experience

3–4 years
(hands -on ownership in DL / LLM / GenAI testing)

Employment Type

Full -time

Role Overview

We are
seeking a hands -on Testing Lead to own quality and documentation for our Deep Learning, LLM, and Vision -Language Model (VLM) products. You will
define how we test, measure, document, and communicate AI quality—working
closely with ML, Engineering, and Product teams in a fast -paced startup
environment.

This role
is ideal for someone who believes clear documentation is as critical as good
testing, especially for non -deterministic AI systems.

What You’ll Do

Own Quality & Documentation End -to -End

• Define testing strategy for LLMs,
  VLMs, and DL pipelines.
  
• Create and maintain clear,
  lightweight documentation covering:
  
• Model testing strategies
  and assumptions
  
• Evaluation metrics and
  acceptance criteria
  
• Known limitations, risks,
  and failure modes
  
• Release readiness and
  quality sign -off
  
• Ensure documentation evolves
  with models, data, and prompts.
  

LLM / GenAI Testing

• Design tests for:
  
• Prompt templates and prompt
  changes
  
• RAG pipelines (retrieval
  quality, grounding, hallucination control)
  
• Multi -turn conversations
  and long -context behaviour
  
• Maintain golden datasets,
  regression test suites, and test result summaries.
  
• Document prompt behaviour,
  edge cases, and known model quirks.
  

Vision & Multimodal Testing

• Test VLMs for image -text
  alignment, OCR, captioning, and reasoning.
  
• Document model performance
  across different image types, quality levels, and domains.
  
• Track and publish model
  behaviour changes between versions.
  

Automation, MLOps & Reporting

• Build Python -based
  automation for evaluation and regression testing.
  
• Integrate tests into CI/CD
  and MLOps pipelines.
  
• Produce readable quality
  reports and dashboards for engineers and leadership.
  
• Monitor and document
  production issues such as model/data drift and degradation.
  

Build a Quality -First Culture

• Establish QA and
  documentation standards that scale with a startup.
  
• Mentor engineers on writing
  testable code and meaningful documentation.
  
• Act as the single source
  of truth for AI quality, testing, and known risks.
  

What we’re looking For

Must -Have

• Strong background in software
  testing with lead or ownership experience.
  
• Hands -on experience testing LLMs,
  DL models, or GenAI systems.
  
• Strong Python skills
  for test automation and data validation.
  
• Proven ability to write clear,
  structured technical documentation.
  
• Understanding of:
  
• Transformer -based models
  and DL workflows
  
• Model evaluation metrics
  and non -deterministic system testing
  
• Comfortable working in
  ambiguity and moving fast in a startup.
  

Nice -to -Have

• Experience with VLMs,
  multimodal models, or computer vision.
  
• Exposure to RAG
  architectures, vector databases, and embeddings.
  
• Familiarity with tools like
  LangChain, LlamaIndex, MLflow, or similar.
  
• Experience documenting AI
  risks, limitations, or compliance requirements.