AI Engineer: Evaluation & Alignment

This role is about understanding how to evaluate model behavior in realistic scenarios: multi-turn conversations, ambiguous queries, edge-case prompts, or compliance-critical interactions. You’ll design the metrics, evaluation logic, and infrastructure that helps customers ensure their Gen AI applications are robust, reliable, and compliant.

What You’ll Do

• Develop evaluation strategies tailored to LLM-based systems, especially in production-like environments with nuanced conversational behavior or task-specific requirements.

• Design and refine metrics that capture key quality dimensions—robustness (handling edge cases and adversarial prompts), reliability (predictable, consistent behavior), and compliance (alignment with user, business, or regulatory expectations).

• Build automated pipelines that evaluate Gen AI application performance across real-world use cases, including prompt variation, context manipulation, and reasoning-based tasks.

• Work across disciplines with engineers and product leads to formalize evaluation goals, define benchmarks, and iterate on evaluation logic as the system evolves.

• Integrate emerging techniques and academic insights (e.g., adaptive testing, behavioral testing, hallucination metrics) into a practical, production-grade evaluation stack.

You’re a Great Fit If You Have:

Core Strengths:

• Experience evaluating natural language systems, especially LLM-based or generative applications (e.g., chatbots, RAG systems, decision support tools).

• Strong intuition for designing meaningful evaluation metrics beyond accuracy—e.g., for measuring factuality, coherence, sensitivity, or policy violations.

• Proficiency in Python and familiarity with automation tools and evaluation libraries (e.g., LangChain, OpenAI Evals, PromptLayer, custom frameworks).

• Deep curiosity for how language models behave in unexpected or high-stakes contexts, and a drive to improve their reliability.

Nice to Have:

• Exposure to Gen AI alignment methods, LLMOps practices, or compliance-related validation (e.g., GDPR-sensitive outputs, safety constraints).

• Familiarity with academic and industry benchmarks (e.g., TruthfulQA, MMLU), and an understanding of their limitations in real-world scenarios.

• Experience working in startup environments, OSS ecosystems, or cross-functional teams tackling novel AI product challenges.

1. Work at the forefront of Gen AI: Collaborate with some of the most innovative companies building LLM applications. Contribute to the trustworthiness of AI by shaping open-source tools that define how Gen AI is tested and validated.
2. Flexible work arrangements: We understand the importance of work-life balance and offer flexible working options to accommodate personal needs and preferences. We have offices in Berlin (AI Campus) and Potsdam (Griebnitzsee).
3. Compensation: We offer salaries and benefits tailored to your experience and qualifications, along with the opportunity to gain ownership in the company.
4. A supportive and collaborative work environment: We foster a culture of teamwork, collaboration, and mutual respect, where every team member is valued and supported in their professional and personal growth.