Senior Quality Analyst

Job Title: AI QA Engineer – LLM & RAG

Location: Hyderabad

Experience: 4+ Years

Employment Type: Full-time

Job Overview

We are looking for a skilled AI QA Engineer to evaluate and enhance the performance of LLM-based applications, RAG pipelines, and AI-driven search systems. The ideal candidate will have strong expertise in AI evaluation frameworks, search quality metrics, and automated testing methodologies.

Key Responsibilities

• Evaluate AI-generated outputs for correctness, coherence, and usefulness
• Apply LLM evaluation techniques using automated and human-aligned metrics
• Assess RAG pipelines, including both retrieval and generation stages
• Use tools like Ragas to benchmark and improve system performance
• Evaluate search and retrieval quality using precision, recall, and ranking metrics
• Perform semantic similarity testing to validate user intent matching
• Design and execute large-scale query-based evaluation experiments
• Build automated workflows for AI evaluation and testing
• Analyze and compare outputs across different models and configurations
• Evaluate conversational AI systems for context handling and response consistency
• Define and track AI performance metrics (relevance, ranking quality, factual accuracy)
• Create benchmark datasets and ground truth for evaluation

Required Skills & Qualifications

• 4+ years of experience in AI/ML QA, LLM evaluation, or AI system testing
• Hands-on experience with RAG evaluation frameworks (Ragas preferred)
• Strong understanding of search/retrieval evaluation metrics
• Experience with NLP evaluation techniques and semantic similarity
• Strong analytical and problem-solving skills
• Experience in automation and large-scale testing workflows

Good to Have

• Experience with vector databases and embeddings
• Knowledge of prompt engineering and LLMOps
• Familiarity with Python, LangChain, or similar AI frameworks