QA Engineer

We're seeking an innovative QA Engineer to join our team. This role goes beyond traditional testing - you'll be at the forefront of using automation frameworks and AI-powered tools to ensure the quality, reliability, and performance of our AI solutions. You'll work closely with AI Integration engineers and software developers to build robust testing strategies that handle the unique challenges of AI systems, including model validation, data quality, and non-deterministic outputs.

Key Responsibilities

Test Automation Development

• Design, develop, and maintain automated test frameworks using Python and modern testing tools


• Build end-to-end test suites for APIs, data pipelines, and AI model/MCP endpoints


• Implement continuous testing practices integrated with CI/CD pipelines


• Create reusable test libraries and utilities to accelerate testing across projects

AI-Assisted Testing

• Leverage AI tools (like GitHub Copilot, ChatGPT, or dedicated test generation tools) to accelerate test case creation and code reviews


• Explore and implement AI-powered testing solutions for test data generation, visual testing, and anomaly detection


• Experiment with automated test scenario generation using LLMs


• Use AI tools for log analysis and defect pattern recognition

AI-Specific Testing

• Develop testing strategies for LLMs including performance, accuracy, and bias detection


• Test data quality, feature engineering pipelines, and model training workflows


• Validate model output and monitor for model drift in production


• Create test datasets that cover edge cases and ensure model robustness

Quality Strategy & Collaboration

• Define and implement QA best practices and quality metrics for AI products


• Collaborate with developers to identify testability requirements early in the development cycle


• Perform code reviews focused on test coverage and quality


• Document test strategies, test cases, and quality reports

Performance & Security Testing

• Conduct load and performance testing for AI inference endpoints


• Identify bottlenecks in data processing and model serving infrastructure


• Participate in security testing activities focusing on data privacy and model security

Required Qualifications

Education & Experience

• Bachelor's degree in Computer Science, Engineering, or related field


• 3-5 years of experience in software quality assurance and test automation


• Proven experience building and maintaining automation frameworks from scratch


• Experience testing cloud-based applications and APIs


• Experience testing machine learning models, agentic data flows and data-intensive applications

Technical Skills

• Strong proficiency in Python for test automation


• Hands-on experience with testing frameworks: pytest, unittest, Selenium, or similar


• Experience with API testing tools: Postman, REST Assured, or Python requests


• Knowledge of CI/CD tools: GitLab CI, GitHub Actions


• Familiarity with version control systems (Git)


• Understanding of SQL and database testing (incl. Data Lake architectures)


• Experience with containerization (Docker) and orchestration tools


• Experience with AI-powered testing tools or test generation platforms


• Experience with cloud platforms: AWS, Azure, or GCP


• Understanding of microservices architecture and distributed systems


• Experience with workflow orchestration tools (Airflow, Temporal, Prefect, n8n)


• Experience mentoring junior QA engineers

What You'll Work With:

Languages: Python, SQL, JavaScript/TypeScript, YAML
Data Tools: Airflow, dbt, Spark, Polars, pyArrow
APIs & Integration: FastAPI, GraphQL, Kafka, Redis, Javascript/Typescript, WASM
AI/ML: OpenAI API, Pydantic AI, Anthropic Claude, LangChain, vector databases, MCP protocol
Cloud: AWS (Lambda, S3, RDS, Bedrock, SageMaker) or equivalent in GCP/Azure
Infrastructure: Docker, Kubernetes, Terraform, GitHub Action

As part of our commitment to quality and excellence, Medlytix will continue to maintain a safe and healthy environment for you by requiring all applicants to submit to a criminal history check and those tentatively selected for a position to submit to screening for illegal drug use prior to appointment for a job. In addition, applicants may be screened for ability to perform essential functions of some positions.