AI Quality Engineer

Testing AI is fundamentally different from testing traditional software. Outputs are non-deterministic, correctness is often a spectrum, and failures like hallucinations, drift, and prompt injection don't show up in standard test cases.

We're looking for an AI Quality Engineer to build automated testing frameworks, evaluation pipelines, and quality infrastructure for AI agents, agentic workflows, and Foundation AI capabilities. This is a hands-on engineering role focused on defining, measuring, and improving AI system quality at scale.

Key Responsibilities

• Define testing strategies for AI agents, conversational interfaces, and agentic workflows
• Design behavioral test suites for non-deterministic AI outputs
• Build evaluation frameworks to measure groundedness, factuality, relevance, and task completion
• Identify and test AI-specific failure modes including hallucinations, prompt injection, context limitations, and drift
• Develop testing approaches for RAG, function calling, human-in-the-loop, and autonomous workflows
• Build automated evaluation pipelines integrated into CI/CD
• Create test harnesses, regression suites, and reproducible test scenarios for LLM-based systems
• Develop monitoring and alerting for production agent quality, accuracy, latency, and error rates
• Build and maintain evaluation datasets, quality metrics, and reporting dashboards
• Partner with AI engineers and analysts to embed quality throughout the development lifecycle

What We're Looking For

• 3–7 years of Software Engineering, SDET, or Quality Engineering experience
• Strong Python and/or TypeScript skills
• Experience building automation frameworks and CI/CD pipelines
• Understanding of AI/LLM systems and evaluation methodologies
• Strong analytical mindset with a passion for solving complex quality challenges

Preferred Experience

• DeepEval, Ragas, Prompt Testing, RAG Evaluation
• Azure AI Services, Copilot Studio, Power Platform
• Load and performance testing for AI applications