AI Quality Analyst
Role Summary
Ensure reliability, accuracy, and safety of AI/Agentic systems through structured evaluation and testing.
Key Responsibilities
- Design test cases for agent behavior, workflows, and edge cases
- Validate outputs for correctness, hallucination, bias, and compliance
- Build automated and manual evaluation pipelines
- Define quality metrics (accuracy, grounding, latency, safety)
- Perform regression testing for prompts, models, and workflows
- Collaborate with engineers to improve prompts and system responses
Must-Have Skills
- 3–6 years in QA/testing with exposure to AI systems
- Experience validating LLM outputs and prompt behavior
- Strong analytical and debugging skills
- Familiarity with evaluation techniques (human + automated)
Good to Have
- Health insurance domain (policy/claims validation scenarios)
- Experience with AI safety and governance
