Company: WillWare Technologies
Role: Gen AI QA Lead
Experience: 6+ Years
Location: Bangalore, Hyderabad, Pune
Work Mode: Hybrid
Position Summary:
As a GenAI Test Lead, you will define and operationalize quality for AI systems, bridging traditional QA with AI engineering. You will design scalable evaluation frameworks (EvalOps) to measure groundedness, factual accuracy, coherence, and RAG performance, ensuring our AI systems are reliable, aligned, and production-ready.
This role requires treating LLM outputs as measurable signals—building data-driven validation systems that continuously improve model performance and user experience.
Key responsibilities and duties -
- Define evaluation metrics for LLM systems (groundedness, hallucination rate, task success, RAG accuracy)
- Design and implement automated EvalOps pipelines for continuous model validation
- Establish acceptance criteria and quality benchmarks for AI releases
- Build scalable test frameworks for LLM-based systems using tools like DeepEval, pytest, or custom pipelines
- Automate validation of prompts, responses, and agent workflows
- Integrate AI testing into CI/CD pipelines
- Analyze LLM outputs as structured data to identify failure patterns
- Validate data quality, retrieval accuracy, and grounding in RAG systems
- Design synthetic and real-world test datasets
- Collaborate with AI engineers, product teams, and clients to define quality standards
- Drive alignment across distributed/onshore-offshore teams
- Provide insights on model performance and improvement areas
- Evaluate prompt strategies, agent behaviors, and user interaction flows
- Ensure outputs align with business context and user expectations
- Rapidly adapt to domain-specific requirements
- Experience with Rational Tool Suite, JIRA, or similar test management tools.
- Experience in using/setting up continuous integration testing and experience with any DevOps tools
- Develop automation tests as part of the Eval Framework using frameworks like DeepEval, pytest etc
- Strong understanding of Agile principles; experience in SAFe Agile environments.
- Experience testing APIs and data integrations.
- Experience working in an onshore/offshore model.
- Experience in test automation design and execution.
- Strong understanding of different software testing methodologies.
Required Qualifications and Experience:
- Minimum 6+ years of overall experience in software QA.
- Education: Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.
- Problem-Solving: Strong analytical and problem-solving skills with the ability to troubleshoot complex, distributed AI systems.
- Communication: Excellent communication skills to articulate technical findings and development progress effectively.
