- Posted a day ago
- Be among the first 30 applicants
Job Description
We are looking an experienced AI Test Engineer to drive quality assurance for AI-powered applications, including LLM models, conversational bots, autonomous agents, and MCP-enabled systems. The ideal candidate will have strong expertise in Playwright with .NET, API/UI automation, AI/LLM validation, and CI/CD automation. The role involves building scalable test frameworks, validating AI responses, testing agent workflows, and ensuring the reliability, security, and performance of GenAI solutions.
Role :AI Test Engineer
Location: PAN India
Experience Range: 5 to 12years
Key Responsibilities:
- Design and execute test strategies for LLM-powered applications, AI chatbots, conversational AI, and autonomous agents.
- Develop and maintain Playwright (.NET) automation frameworks for UI and API testing.
- Validate AI model outputs for:
- Accuracy
- Relevance
- Hallucinations
- Consistency
- Safety and Guardrails
- Test MCP (Model Context Protocol) endpoints and MCP-enabled agent workflows.
- Automate API testing and integrate test suites into CI/CD pipelines.
- Create test datasets, evaluation metrics, and AI validation frameworks.
- Perform regression, functional, integration, and end-to-end testing for AI systems.
- Collaborate with developers, data scientists, and product teams to ensure quality delivery.
- Monitor and troubleshoot AI system failures, agent interactions, and model behavior.
- Implement quality gates within Azure DevOps/GitHub Actions/Jenkins pipelines.
Required Skills:
AI / GenAI Testing
- LLM Testing
- Prompt Validation
- AI Response Validation
- Conversational Bot Testing
- Autonomous Agent Testing
- MCP (Model Context Protocol)
- Hallucination Detection
- AI Safety & Guardrail Testing
- RAG Validation
Automation Testing
- Playwright with .NET (C#)
- UI Automation
- API Automation
- Postman / REST APIs
- Selenium (Good to Have)
Programming
- C#
- .NET Core / ASP.NET
- JavaScript / TypeScript (Good to Have)
DevOps & CI/CD
- Azure DevOps
- GitHub Actions
- Jenkins
- Docker (Preferred)
Other Skills
- Agile/Scrum
- SQL Databases
- Test Framework Design
- Version Control (Git)
More Info
Key Skills
LLM Testing
Prompt Validation
Conversational Bot Testing
GitHub Actions
Autonomous Agent Testing
AI Safety Guardrail Testing
RAG Validation
Playwright with .NET
Postman REST APIs
Hallucination Detection
MCP Model Context Protocol
AI GenAI Testing
AI Response Validation
About Company
