Role: AI Tester
Location: Indore
Experience: 6+ Yrs.
Company's Website: http://www.yash.com/
- Skillset: Gen AI + LLM + Agentic AI
- Key Responsibilities
- Design and implement test strategies for text-based AI agents and conversational systems
- Understands LLM responses, reasoning accuracy, and hallucination risks AI Response Validation.
- Manually verify that AI agent responses are accurate, non-toxic, and free of hallucinations.
- Functional & Agentic Testing: Test the agent's ability to call external tools (MCP, APIs) and execute complex workflows correctly.
- Scenario Testing: Develop and execute manual test cases for diverse, edge-case user queries, including adversarial testing (prompt injection).
- Defect Documentation: Identify, document, and report bugs in AI reasoning, UI, and performance in tools like JIRA.
- UAT & Usability: Perform User Acceptance Testing to ensure the AI interaction is intuitive and meets end-user needs.
- Dataset Curation: Collaborate with developers to create high-quality datasets for training and testing.
- Required Skills & Qualifications:
- Testing Expertise: 4+ years of experience in functional, regression, and usability testing.
- Additional weightage will be given to candidates worked on AI Product Testing.
- AI Familiarity: Understanding of LLM behavior, chatbots, or agentic systems.
- API Testing: Hands-on experience with tools like Postman to validate AI API calls.
- Analytical Skills: Strong ability to analyze unstructured conversational data and identify patterns.
- Technical Aptitude: Basic knowledge of Python or SQL is a plus for analyzing logs.
- Must-Have Skills AI Testing & Evaluation
- Experience testing AI chatbots Prompt testing and response evaluation Hallucination detection and output validation Testing RAG pipelines and knowledge retrieval
