AGENTIC AI

MODEL VALIDATOR

Eval set development - ability to benchmark agent performance through reasoning paths

Adversarial testing - ability to break agent by giving it conflicting instructions etc

Scholastic regression testing - Measure variance in agent behavior.

Tool call validations - Agent calls the correct external APIs and databases

Ability to review thought chains and identify where agent logic diverged from BRD

Must have knowledge of applying judge LLMs to grade outputs

Python and framework - Proficiency in DeepEval, Langsmith etc

Ability to do semantic debugging - Look at agent's thought trace

Screening Criteria

SDETs as they have coding background to testing - They can develop evals

Good knowledge of data / SQL based testing etc

Domain background is added advantage for such roles