Responsibilities:
• Build API and integration test coverage for routers and database flows.
• Design and implement E2E tests with Playwright for critical user flows.
• Create stable mocking and fixture strategies (MSW, pytest fixtures, predefined test data).
• Configure quality gates in CI for linting, unit/integration/E2E tests, code coverage, and reporting.
• Extend coverage into non-test UI and workflow areas and prevent regressions.
• Build an LLM evaluation discipline with golden datasets and measurable quality gates.
• Track reliability of model outputs, detect drift, and condition releases on quality/stability thresholds.
• Improve test maintainability, eliminate flakiness, and optimize runtimes.
Qualifications:
• Strong Python testing skills with pytest.
• Strong frontend testing skills.
• Experience building and maintaining E2E suites with Playwright/Selenium.
• Experience in API testing (REST, schema validation, authentication/error scenarios).
• Experience with SQL/PostgreSQL testing and test data management.
• Experience with CI/CD pipelines (GitHub Actions, Azure Pipelines or Jenkins).
• Practical understanding of testing non-deterministic outputs of LLM models.
• A plus
• Experience in performance and benchmark testing.
• Experience in WebSocket testing.
• Knowledge of security and dependency scanning.
• Experience in legal-tech, fintech or other regulated domains.
• Familiarity with LLM quality metrics (schema validity, hallucination tracking, consistency).
• Tech Stack
• Backend Testing: pytest, pytest-asyncio, coverage.py
• E2E: Playwright
• CI/CD: GitHub Actions
• Data: PostgreSQL, pgvector
• Application Stack Tested: FastAPI, React, TypeScript, Keycloak, Azure OpenAI
*We are an equal opportunity employer and value diversity. All employment decisions are made without regard to age, gender, disability, race, ethnicity, religion, sexual orientation, or any other protected characteristic. We encourage applicants from all backgrounds to apply.*