Lead AI QA Engineer – Manual & Functional Testing
Project Foundry Resourcing Services
About the Role
We are seeking an experienced Lead AI QA Engineer to take ownership of quality assurance across a fast-paced, multi-agent LLM product.
This is a hands-on leadership role responsible for defining the QA approach, improving test coverage, coordinating testing activity and ensuring the quality, accuracy and reliability of AI-driven functionality.
The role will retain a strong focus on manual testing, functional testing, front-end validation, answer accuracy and source-data alignment, while providing technical direction across the wider QA function.
The successful candidate will lead test strategy, planning and quality standards, prioritise activity based on risk and business impact, and ensure AI-generated responses are validated against the correct underlying data sources.
They will work closely with developers, data scientists, AI engineers, product managers and delivery teams to identify risks, drive defects through to resolution and provide a clear view of release readiness.
Key Responsibilities
QA Leadership & Test Strategy
- Own and continuously improve the QA and testing strategy for the AI Assistant and associated product functionality.
- Define testing standards, processes, quality gates and release-readiness criteria.
- Lead QA planning and prioritisation across multiple workstreams and releases.
- Apply risk-based testing to ensure high-impact areas receive appropriate coverage.
- Provide technical guidance and support to QA Engineers and Testers.
- Review test plans, test cases, defect reports and test evidence for quality and completeness.
- Identify gaps in QA processes and implement practical improvements.
- Act as the primary QA point of contact for engineering, product and delivery teams.
- Provide clear recommendations on release readiness and outstanding quality risks.
Manual, Functional & Front-End Testing
- Lead the creation and execution of detailed manual test plans and test cases covering front-end UI, conversational flows and core product functionality.
- Perform and oversee functional, regression, exploratory, smoke, usability and release testing.
- Validate user journeys, prompts, responses, data retrieval, workflows and system behaviour across a wide range of scenarios.
- Test front-end screens, navigation, forms, conversational interfaces, business rules, permissions and access controls.
- Ensure defects and test findings are clearly documented, reproducible and prioritised in Jira or equivalent tools.
- Work directly with developers to reproduce complex defects and validate fixes.
- Maintain clear test evidence, execution results and release-validation documentation.
- Coordinate cross-browser, regression and release testing across new functionality and impacted areas.
AI Response Validation & Accuracy Testing
- Lead validation of LLM-generated responses against structured and unstructured source data.
- Ensure responses are accurate, complete, relevant and grounded in the correct information.
- Define repeatable methods for testing answer accuracy, source alignment and consistency.
- Identify and escalate hallucinations, incorrect responses, source mismatches, missing context, reasoning gaps and inconsistent outputs.
- Develop scenarios covering realistic user behaviour, adversarial prompts, edge cases and complex business use cases.
- Analyse recurring AI quality issues and work with engineering and data teams to identify root causes.
- Support the development of quality metrics, evaluation frameworks and acceptance criteria for AI-generated outputs.
Automation, Performance & Security
- Define where automation can add value while retaining strong manual validation for complex AI behaviours.
- Support automated UI, API, regression and performance testing where appropriate.
- Improve reusable test assets, regression suites and overall test efficiency.
- Coordinate testing of response times, system performance and behaviour under different workloads.
- Work with engineering and security teams to validate privacy, permissions, access controls and compliance requirements.
- Test scenarios where users could receive incorrect, unauthorised, sensitive or inappropriate information.
- Validate AI behaviour across different user roles and data-access scenarios.
Defect Management & Release Quality
- Own the QA view of defect severity, priority and overall product quality.
- Lead defect triage with engineering and product teams.
- Ensure critical and high-impact issues are investigated and resolved appropriately.
- Monitor defect trends and identify recurring quality issues or systemic weaknesses.
- Coordinate release validation and provide clear quality recommendations ahead of deployment.
- Support post-release validation and rapid assessment of production issues.
- Communicate outstanding quality risks clearly before releases are approved.
Collaboration & Reporting
- Work closely with Developers, Data Scientists, AI Engineers, DevOps, Product Managers and delivery teams.
- Contribute to sprint planning, stand-ups, retrospectives, backlog refinement and release-readiness discussions.
- Represent QA during product design, technical discussions and feature development.
- Communicate testing progress, blockers, quality risks and release concerns to technical and non-technical stakeholders.
- Provide structured reporting on test progress, defect status, quality trends and release readiness.
- Promote a strong quality culture across engineering and product teams.
Required Skills & Experience
Must Have
- Significant software testing and QA experience, including Senior, Lead or Test Lead responsibility.
- Strong hands-on experience in manual and functional testing.
- Proven experience testing AI, LLM, chatbot, virtual assistant or NLP-driven products.
- Experience leading QA across complex software products or multiple development workstreams.
- Strong knowledge of QA strategy, risk-based testing, test planning and release-quality management.
- Strong understanding of functional, regression, exploratory, smoke, usability and front-end testing.
- Experience creating and reviewing test plans, test cases, scripts, test evidence and defect reports.
- Proven ability to validate AI-generated responses against underlying source data.
- Strong ability to identify hallucinations, inaccuracies, source mismatches, inconsistent outputs and missing context.
- Good understanding of LLMs, NLP, retrieval-based AI systems and AI response validation.
- Experience leading defect triage and investigating complex issues with engineering teams.
- Experience providing release-readiness assessments and communicating quality risks.
- Strong attention to detail, structured problem-solving and troubleshooting skills.
- Ability to prioritise QA activity based on business impact, technical risk and release urgency.
- Experience working in fast-paced environments with evolving requirements.
- Strong written and verbal communication skills.
- Ability to mentor and provide technical guidance to QA Engineers or Testers.
- Comfortable working across engineering, AI, data, product and delivery teams.
Desirable Skills
- Experience using Jira or similar tools for defect tracking, test management and reporting.
- Experience improving QA processes within an AI or software product environment.
- Exposure to automated UI, API or regression testing frameworks.
- Experience testing data-driven, API-enabled or multi-agent AI products.
- Experience with RAG, vector databases, embeddings or similar AI architectures.
- Experience developing LLM evaluation frameworks or tracking quality metrics such as accuracy, groundedness, relevance and consistency.
- Experience supporting performance/load testing, UAT, production validation and post-release defect triage.
- Familiarity with privacy, security, permissions and access-control considerations in AI or data-driven applications.
- Experience working within Agile software development environments.
