Responsibilities
- Build and own eval infrastructure for production AI systems, golden datasets, regression suites, and LLM-as-judge harnesses.
- Own API and backend test automation across microservices and async pipelines, not the UI layer.
- Write, co-write, and review test design documentation; drive code review for automation frameworks.
- Drive the design/code review process for test automation, seeking and providing constructive criticism.
- Lead observability design so anyone can answer did the AI get worse this week with a chart, not a gut feel.
- Own quality communication across sprint and release cycles.
Requirements
- 2-5 years QA/SDET with a minimum of 1.5-2 years of backend/API testing.
- Strong coding in Java, Python, or TypeScript.
- Hands-on with REST Assured, Pytest, or equivalent coded API frameworks.
- Understands microservices, async systems, and event-driven architecture.
- Builds automation frameworks from scratch, not just uses tools.
- CI/CD integration experience.
- Systems thinker - reasons about failure modes, retries, latency, contracts.
- AI/LLM testing is a strong plus, RAG, hallucination detection, and eval frameworks.
- Experience working with Web and API Testing - both manual and automation.
- Performance testing exposure preferred (JMeter, k6) and Agile experience.
This job was posted by S M Nandakishore from CAW Studios.
