Data Scientist - AI evaluation

 OVERVIEW
We are seeking an experienced Data Scientist – AI Evaluation Analytics with a strong background in data science, analytics, and statistical methods to support the client's efforts to improve the measurement, reliability, and interpretation of AI system performance. The successful candidate will work closely with cross-functional partners — including AI research, engineering, and product teams — to design evaluation metrics, validate data quality, and generate actionable insights that drive improvements in AI system effectiveness.

This role is ideal for a proactive, analytically minded professional who is comfortable working with large datasets, experimentation frameworks, and evolving evaluation methodologies in a fast-paced AI environment.

Location: Heredia, CR (Hybrid)
Contract Length: 6+ Months (Extension or FTE Conversion Possible)
Hours: Standard Business Hours, Local Time Zone
Experience Level: Intermediate (3–5 years)

• RESPONSIBILITIES
  Support the development and refinement of evaluation metrics, scoring approaches, and benchmark methodologies for AI systems.


• Validate datasets, data quality criteria, and evaluation inputs to improve the reliability of results.


• Analyze evaluation data, performance trends, and measurement signals to generate actionable insights.


• Apply statistical analysis and experimentation techniques to assess AI system effectiveness.


• Develop analytical approaches to identify trends, patterns, and opportunities for improvement.


• Collaborate with AI research, engineering, and product teams to ensure evaluation findings translate into meaningful system improvements.


• Prepare clear, concise documentation and reports summarizing analytical findings and recommendations for stakeholders.


• Maintain data integrity and ensure evaluation processes align with internal quality and reliability standards.

• Proficiency in Python, SQL, and data analysis techniques.


• Experience working with large datasets and extracting meaningful, actionable insights.


• Strong communication skills with the ability to explain analytical results clearly to both technical and non-technical stakeholders.


• Ability to manage multiple priorities in a fast-paced environment with minimal supervision.

Preferred

• Experience evaluating AI/ML systems and agentic AI applications.


• Experience with experimentation, benchmarking, or performance analysis.


• Experience working with software engineering, productivity, or quality metrics.

Notes for Applicants
Resumes must clearly demonstrate hands-on experience with data analysis, statistical methods, and Python/SQL, ideally with examples involving large datasets or AI/ML evaluation.