QA Architect
About Nous
Nous Infosystems is a CMMI Level 5 and ISO 9001:2000 certified global information Technology Company with expertise in providing quality software solutions and IT-enabled support services to a wide range of industries. Nous was incorporated in 1996 with the mission of providing the highest quality software development services to clients around the world. Over the years, we have successfully broadened our service offerings to include consultancy services, software and hardware testing services, and a comprehensive range of IT enabled services.
We are on the lookout for QA Architect with the following skills.
Job Title: AI Test Lead / AI Test Architect
Experience: 814 Years
Location: Bangalore
Joining: Immediate / Short Notice Preferred
Role Overview
We are looking for a highly skilled AI Test Lead / AI Test Architect to drive the quality assurance strategy for GenAI, LLM, and RAG-based applications. This role requires deep hands-on expertise in validating AI systems, designing scalable AI test frameworks, and ensuring accuracy, reliability, safety, and performance of AI-driven solutions in enterprise environments.
Key Responsibilities
- Lead and define end-to-end QA strategy for AI/ML, GenAI, LLM, and RAG-based applications.
- Design and implement test frameworks and automation solutions for validating:
- LLM responses (accuracy, hallucination, bias, toxicity)
- RAG pipelines (retrieval relevance, grounding, faithfulness)
- Prompt engineering effectiveness and prompt versioning
- Establish AI testing methodologies including functional, non-functional, regression, performance, and reliability testing for AI systems.
- Validate AI models for fairness, bias, explainability, robustness, and safety.
- Implement AI evaluation metrics such as precision/recall, relevance score, grounding score, latency, and response consistency.
- Collaborate closely with AI/ML engineers, data scientists, backend teams, product managers, and DevOps to integrate testing into CI/CD pipelines.
- Drive test data strategy for AI systems including synthetic data generation and edge-case coverage.
- Review AI architecture and proactively identify quality risks and failure points.
- Mentor QA teams and establish AI testing best practices, standards, and governance.
- Support UAT, production validation, and post-deployment monitoring of AI systems.
Required Skills & Experience
- 8+ years of overall QA / Testing experience with 2+ years in QA Lead / Architect role.
- Strong hands-on experience in testing GenAI / LLM-based applications.
- Proven experience validating RAG pipelines (retriever, embeddings, vector databases, reranking).
- Hands-on exposure to LLM testing techniques including:
- Hallucination detection
- Prompt testing & prompt drift
- Response consistency and determinism
- Experience with AI testing frameworks/tools such as:
- RAGAS, LangSmith, DeepEval, TruLens (or equivalent)
- Strong automation experience using Python / Java / JavaScript.
- Experience with API testing tools (Postman, REST Assured, ReadyAPI, Swagger).
- Knowledge of CI/CD pipelines, Git, Docker, and cloud platforms (AWS/Azure/GCP).
- Strong understanding of AI lifecycle, model behavior, and limitations.
Please do share your latest profile to [Confidential Information] with the following details.
Current CTC :-
Expected CTC :-
Minimum Notice period :-
Regards
Team TA
9880370901
