Quality Assurance Engineer (AI)

Join Delphi - Where Innovation meets transformation

At Delphi, we believe in creating an environment where our people thrive. Our hybrid work model empowers you to choose where you workwhether it's from the office, your home, or a mix of bothso you can prioritize what matters most. We are committed to supporting your personal goals, family, and overall well- being while driving transformative results for our clients.

We welcome exceptional talent from anywhere across the globe. Interviews and onboarding are conducted virtually, reflecting our digital-first mindset.

Rooted in the region, we specialize in delivering tailored, impactful solutions in Data, Advanced Analytics and AI, Infrastructure, Cloud Security, and Application Modernization. Whether it's enabling predictive analytics, transforming operations with automation, or driving customer engagement with intelligent platforms, we are the trusted partner for organizations ready to embrace a smarter, more efficient future.

About the Role: Sr Consultant - Quality Assurance

Location: India (Remote)

As an Sr Consultant - Quality Assurance (AI), you will play a critical role in ensuring the performance, reliability, and trustworthiness of advanced Large Language Models (LLMs), sub-models, and multi-agent systems. You will design and execute comprehensive evaluation strategies, leveraging both manual and automated testing to assess model accuracy, consistency, and robustness. With strong expertise in Python and synthetic data generation, you'll simulate real-world and edge-case scenarios to validate model behavior under diverse conditions. Collaborating closely with Data Science and Engineering teams, you will align QA frameworks with model development, utilize cutting-edge evaluation tools and metrics, and maintain detailed documentation and prompt libraries. This role offers an exciting opportunity to shape the quality and performance standards of next-generation AI systems.

What You'll Do:

• Design and execute evaluation strategies for LLMs, sub-models, and multi-agent systems.
• Perform manual and automated testing of AI models, ensuring comprehensive coverage.
• Develop synthetic test data using Python scripting to simulate real-world scenarios and edge cases.
• Ensure boundary condition coverage through automated data generation techniques.
• Prepare and maintain prompt libraries for model evaluation across diverse use cases.
• Evaluate multi-model and multi-agent architectures, ensuring functional and performance benchmarks are met.
• Apply and interpret evaluation metrics (e.g., accuracy, precision, recall, F1-score, Bias check, hallucination, Consistency, Relevance, Fact Check, BLEU, ROUGE, perplexity, etc.) with a clear understanding of their formulas and implications.
• Collaborate closely with Data Science and Engineering teams to align QA strategies with model development.
• Document test plans, test cases, and evaluation reports for stakeholder review and signoff.

What You'll Bring:

• 5+ years of relevant experience in quality assurance and AI/ML evaluation.
• Strong hands-on experience with LLM evaluation techniques.
• Proficiency in Python for scripting, test automation, and synthetic data generation.
• Deep understanding of AI model architecture, including models, sub-models, and agent-based systems.
• Familiarity with prompt engineering and its role in model testing.
• Experience with AI/ML testing frameworks and tools.
• Solid grasp of evaluation metrics and their practical application.
• Excellent analytical, documentation, and communication skills.
• Prior experience in QA for AI/ML products.
• Exposure to tools like RAGAS, DeepEval, Arize Phoenix, OpenAI APIs, or similar.
• Experience with CI/CD pipelines for AI model deployment and testing.
• A sense of humor

What We Offer:

At Delphi, we are dedicated to creating an environment where you can thrive, both professionally and personally. Our competitive compensation package, performance-based incentives, and health benefits are designed to ensure you're well-supported. We believe in your continuous growth and offer company- sponsored certifications, training programs, and skill-building opportunities to help you succeed. We foster a culture of inclusivity and support, with remote work options and a fully supported work-from- home setup to ensure your comfort and productivity. Our positive and inclusive culture includes team activities, wellness and mental health programs to ensure you feel supported.