AI Test Lead
Experience: 7+ Years
Location: Remote
Position Summary
We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands-on exposure to AI/ML, Generative AI, LLM, and RAG-based applications.
The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.
Key Responsibilities
- Lead end-to-end testing activities for AI/ML and Generative AI applications.
- Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
- Design and implement automated testing frameworks for web, API, and AI-based applications.
- Test LLM, RAG, prompt-based applications, chatbots, and AI agents.
- Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
- Develop test datasets and evaluation methodologies for AI/GenAI applications.
- Perform API, functional, regression, integration, and end-to-end testing.
- Collaborate with developers, data scientists, product owners, and business stakeholders.
- Identify and troubleshoot defects and quality issues across application and AI components.
- Integrate automated tests into CI/CD pipelines.
- Mentor QA engineers and provide technical leadership to the testing team.
- Establish quality metrics, reporting, and continuous improvement practices.
- Ensure testing standards and best practices are followed across projects.
Required Skills
- 7+ years of experience in Software Testing / QA / Test Automation.
- Hands-on experience in AI/GenAI/LLM testing.
- Strong experience with Selenium, Playwright, Cypress, or similar automation tools.
- Strong programming skills in Python or Java.
- Experience in API testing using REST, Postman, Rest Assured, or similar tools.
- Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications.
- Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality.
- Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals, or equivalent.
- Good understanding of SQL and database testing.
- Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
- Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.
Good to Have
- Experience testing LangChain / LangGraph applications.
- Knowledge of vector databases and RAG architectures.
- Exposure to cloud platforms such as AWS, Azure, or GCP.
- Knowledge of AI security testing and responsible AI principles.
- Experience in performance and security testing.
- Previous experience leading or mentoring QA teams.
Education
Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
Candidate Profile
The ideal candidate should have strong test leadership and automation experience along with practical exposure to AI/GenAI testing. The ability to work closely with engineering, data science, and product teams and translate AI quality requirements into effective testing strategies is essential.
