Job Description
This is a remote position.
Job Summary
We are seeking an experienced and hands-on
QA Lead AI & GenAI Systems to own and drive quality across next-generation AI-powered products. The ideal candidate will bring deep expertise in traditional QA practices along with strong experience testing
GenAI, LLM-based, and agentic systems in high-scale production environments.
You will define the QA strategy, lead automation initiatives, establish AI-specific quality metrics, and work closely with engineering, ML, and product teams to ensure reliability, safety, and performance of complex AI workflows. This role is critical in shaping quality standards for non-deterministic systems, multi-agent architectures, and retrieval-augmented generation (RAG) pipelines in a fast-paced startup environment.
Responsibilities
- Own and define the end-to-end QA strategy, test planning, and quality metrics for AI and non-AI systems.
- Lead and mentor QA engineers, ensuring best practices in automation, test design, and execution.
- Design and implement automated test frameworks for API, backend, integration, and regression testing.
- Integrate QA processes into CI/CD pipelines to enable continuous testing and rapid feedback loops.
- Collaborate closely with engineering, ML, and product teams to validate functional and non-functional requirements.
- Define and track AI-specific quality metrics including accuracy, relevance, hallucination rate, latency, and consistency.
- Test GenAI / LLM-based systems including hosted and open-source model integrations.
- Validate non-deterministic behaviors, probabilistic outputs, and prompt-based workflows.
- Lead testing for RAG systems, including vector search, embeddings, retrieval accuracy, and response grounding.
- Execute safety, bias, and guardrail testing to ensure responsible AI behavior.
- Support evaluation frameworks such as human-in-the-loop, offline benchmarking, and online experimentation.
- Validate data quality used for model training, fine-tuning, and inference pipelines.
- Test agent workflows involving multi-step reasoning, tool calling, memory/state handling, and orchestration logic.
- Collaborate on prompt engineering, prompt regression testing, and prompt versioning strategies.
Requirements
Essential Skills:
Job
- 812+ years of experience in software QA with 35+ years leading QA teams.
- Strong expertise in API, backend, and integration test automation.
- Hands-on experience with CI/CD pipelines and automated regression testing.
- Proven experience testing GenAI / LLM-based applications in production environments.
- Deep understanding of non-deterministic systems and probabilistic output validation.
- Experience with RAG architectures, embeddings, vector databases, and retrieval quality testing.
- Familiarity with LLMOps / MLOps tools, model monitoring, and evaluation pipelines.
- Experience testing multi-agent systems or agent orchestration frameworks.
- Strong understanding of SDLC, Agile methodologies, and quality governance.
Personal
- Strong leadership and mentoring capabilities.
- Excellent analytical, problem-solving, and decision-making skills.
- Ability to collaborate effectively with cross-functional technical and non-technical teams.
- Strong communication skills with the ability to explain complex QA and AI concepts clearly.
Preferred Skills
Job
- Experience working in startup or fast-paced product development environments.
- Exposure to AI evaluation frameworks, benchmarking techniques, and A/B testing.
- Knowledge of prompt engineering best practices, prompt regression, and version control.
- Experience testing high-scale, distributed, or cloud-native AI systems.
- Familiarity with tools such as Jira, Confluence, TestRail, or similar QA management platforms.
Personal
- Proactive mindset with a strong sense of ownership and accountability.
- Ability to work under tight timelines and evolving requirements.
- Strong attention to detail while balancing speed and quality
- Passion for building reliable, responsible, and scalable AI products.
Other Relevant Information
- Bachelor's degree in Engineering (BE/B.Tech CS/IT) or Master's degree in Computer Applications (MCA) or equivalent qualification.
- Ability to work independently and collaboratively in a global, fast-paced environment.
Benefits
- This role offers the flexibility of working remotely in India.
LeewayHertz is an equal opportunity employer and does not discriminate based on race, colour, religion, sex, age, disability, national origin, sexual orientation, gender identity, or any other protected status. We encourage a diverse range of applicants.
check(event) ; career-website-detail-template-2 => apply(record.id,meta) mousedown=lyte-button => check(event) final-style=background-color:#6875E2;border-color:#6875E2;color:white; final-class=lyte-button lyteBackgroundColorBtn lyteSuccess lyte-rendered=>
