Data Annotation Lead

AI Data Quality Lead – Robotics & Video Annotation | Remote

About the Role

We are looking for an experienced AI Data Quality Lead to own the quality standards for egocentric training data used to develop and evaluate physical-AI and robotics models.

You will be responsible for building and maintaining the golden sets, annotation SOPs, rubrics, calibration systems, quality frameworks, and audit processes that ensure high-quality ground-truth data. You will also work closely with engineering and product teams to develop annotation workflows for hand tracking, object identification, and action labeling.

This is a ground-floor opportunity to build a 0→1 data quality system at a fast-growing, well-funded startup. You will not simply inherit an existing process—you will help establish the quality standards, workflows, and systems from the ground up.

Key Responsibilities

• 
  

  Own the end-to-end quality framework for egocentric training data, including:

  
  
  
  
  • 
    

    Golden and benchmark datasets

    
    
  
  
  • 
    

    Acceptance criteria

    
    
  
  
  • 
    

    Frame-level tolerance thresholds

    
    
  
  
  • 
    

    Sampling strategies

    
    
  
  
  • 
    

    Audit and quality-control loops

    
    
  
  
  
  


• 
  

  Author, maintain, and version annotation SOPs, rubrics, and edge-case guidelines as project specifications evolve.

  
  


• 
  

  Translate ambiguous requirements into precise, actionable instructions for annotators and reviewers.

  
  


• 
  

  Establish and manage annotator training, certification, and calibration programs.

  
  


• 
  

  Run calibration sessions and adjudicate disagreements between annotators.

  
  


• 
  

  Measure and report quality using metrics such as:

  
  
  
  
  • 
    

    Inter-Annotator Agreement (IAA)

    
    
  
  
  • 
    

    Kappa/alpha

    
    
  
  
  • 
    

    Tolerance-based agreement for temporal boundaries

    
    
  
  
  • 
    

    Acceptance/rejection rates

    
    
  
  
  • 
    

    Rework rates

    
    
  
  
  • 
    

    Cost per accepted hour

    
    
  
  
  
  


• 
  

  Identify root causes of quality issues and determine whether they result from guideline ambiguity, annotation errors, or workflow issues.

  
  


• 
  

  Design human-in-the-loop workflows where AI/model-generated labels are reviewed, corrected, and escalated by human annotators.

  
  


• 
  

  Audit auto-generated labels and improve annotation throughput without compromising quality.

  
  


• 
  

  Partner with engineering and product teams to develop new annotation workflows for:

  
  
  
  
  • 
    

    Hand tracking

    
    
  
  
  • 
    

    Object identification

    
    
  
  
  • 
    

    Action labeling

    
    
  
  
  
  


• 
  

  Define label schemas, annotation-tool requirements, QC dashboards, and pilot-validation processes.

  
  


• 
  

  Validate annotation workflows through pilot programs before scaling them into production.

  
  


• 
  

  Work with ML teams to understand how annotation quality impacts model performance.

  
  


• 
  

  Contribute to tooling and workflow improvements where required.

  
  

Required Skills & Qualifications

• 
  

  5+ years of hands-on video annotation experience, including at least 1+ year working with egocentric/first-person video for robotics or embodied AI.

  
  


• 
  

  3+ years of experience in a lead or QA capacity, including annotator calibration, disagreement adjudication, and ownership of annotation guidelines.

  
  


• 
  

  Demonstrated experience building 0→1 annotation programs and taking them from pilot through production.

  
  


• 
  

  Strong command of annotation-quality methodologies, including:

  
  
  
  
  • 
    

    IAA frameworks

    
    
  
  
  • 
    

    Kappa/alpha

    
    
  
  
  • 
    

    Gold-set creation

    
    
  
  
  • 
    

    Sampling strategies

    
    
  
  
  • 
    

    Tolerance thresholds

    
    
  
  
  • 
    

    Acceptance criteria

    
    
  
  
  
  


• 
  

  Ability to design quality systems and explain the practical difference between quality targets such as 95% and 99% accuracy.

  
  


• 
  

  Proven experience authoring, rather than simply following, annotation SOPs and rubrics.

  
  


• 
  

  Ability to translate ambiguous specifications into clear and actionable annotation guidelines.

  
  


• 
  

  Hands-on experience with at least 2 professional annotation platforms, such as:

  
  
  
  
  • 
    

    CVAT

    
    
  
  
  • 
    

    Labelbox

    
    
  
  
  • 
    

    Encord

    
    
  
  
  • 
    

    V7

    
    
  
  
  • 
    

    Label Studio

    
    
  
  
  • 
    

    Equivalent annotation tools

    
    
  
  
  
  


• 
  

  Experience configuring label schemas and QA workflows within annotation platforms.

  
  


• 
  

  Experience auditing model-generated or automatically generated labels.

  
  


• 
  

  Experience designing human-in-the-loop annotation workflows.

  
  


• 
  

  Strong written and verbal English communication skills.

  
  


• 
  

  Strong analytical skills and proficiency with Excel and/or Google Sheets for quality reporting.

  
  


• 
  

  Strong understanding of ML concepts and how annotation quality affects model performance.

  
  


• 
  

  Data-driven and process-oriented mindset with strong ownership.

  
  


• 
  

  Comfortable working in ambiguous, fast-moving, startup environments.

  
  


• 
  

  Strong ability to work collaboratively with cross-functional and remote teams.

  
  

Preferred / Bonus Qualifications

• 
  

  Background in robotics, mechanical engineering, mechatronics, or computer vision engineering.

  
  


• 
  

  Experience working at a leading robotics company, AI/robotics startup, or similar project.

  
  


• 
  

  Experience managing distributed or global annotator workforces or external annotation vendors.

  
  


• 
  

  Experience with multimodal annotation, including:

  
  
  
  
  • 
    

    3D/depth data

    
    
  
  
  • 
    

    Joint pose

    
    
  
  
  • 
    

    Grasp outcome classification

    
    
  
  
  
  


• 
  

  Experience training or fine-tuning autolabeling models.

  
  


• 
  

  Experience partnering closely with ML teams responsible for automated labeling.

  
  


• 
  

  Experience building annotation tooling or working closely with annotation-tooling engineering teams.

  
  


• 
  

  Experience with physical AI, embodied AI, computer vision, or robotics datasets.

  
  

Why This Role?

The quality bar is the product.

The ground-truth data produced through this role will become a benchmark against which physical-AI models are measured. You will have direct ownership of the systems that determine whether that data meets the required quality standards.

This is a ground-floor opportunity at a fast-growing, well-funded startup where you will build the quality system from scratch rather than inherit an established process.

Role Highlights

• 
  

  Role: AI Data Quality Lead – Robotics & Video Annotation

  
  


• 
  

  Work Type: Remote

  
  


• 
  

  Engagement: [Contract / Full-Time – specify as applicable]

  
  


• 
  

  Focus: AI Training Data, Robotics, Embodied AI & Video Annotation

  
  


• 
  

  Experience: 5+ years relevant experience