Quality Assurance Analyst

Exp-6-8 Years

Notice-Immediate-serving 10 Days

Location-Pune (Hinjewadi-Phase 3)

Mode-2 Days from Office

Key Responsibilitie

• s
  Design and execute test cases for LLM agents, RAG pipelines, agentic workflows, and AI-assisted decision too
• lsValidate AI outputs against ground truth using structured accuracy scoring (NAICS, risk exposure flags, hazard group mappin
• g)Detect hallucinations, reasoning gaps, source fabrication, and misattributions in model-generated conte
• ntRun multi-model comparative testing across GPT, Claude, Gemini, and Perplexity — evaluating accuracy, latency, and output completene
• ssTest prompt versions iteratively and track accuracy changes across prompt cycl
• esValidate citation accuracy, document ingestion pipelines, and cross-document context handli
• ngDesign edge case and negative tests for AI-specific failure modes — content filter triggers, tool call limits, missing documents, and incomplete synthes
• isPerform regression testing after model upgrades, prompt changes, and backend fixes, and maintain structured QA sign-off in JI

RA
What Makes This Role Different from Traditional

• QAYou evaluate whether an AI is reasoning correctly — not just whether the UI behaves as expec
• tedYou build evaluation rubrics for non-deterministic outputs and apply LLM-as-a-Judge techniques to score quality at sc
• aleYou treat every model or prompt change as a potential accuracy regression, not just a functional
• oneYou understand that in live AI systems, a passing test today does not guarantee a passing test tomor

row
Required Skill

• sets7–8 years of QA experience with minimum 2 years in Generative AI / LLM-based proj
• ectsHands-on experience testing chatbots, RAG systems, or agentic AI pipel
• inesProven ability to perform ground truth validation and detect hallucinations and reasoning fail
• uresFamiliarity with multi-model evaluation, prompt-aware testing, and JIRA-based defect repor

tingPreferred Skill

• setsBackground in insurance or regulated industries; exposure to underwriting or risk classification conc
• eptsFamiliarity with Azure OpenAI, AWS Bedrock, or SharePoint-integrated AI environm
• entsKnowledge of AI governance, content filtering, and PII redaction valida