AI/ML Engineer (Contract)

About Us

Deccan AI Experts is a pioneering AI company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in high-quality human-curated data, AI-first operations, and advanced AI evaluation systems. Our global network of financial crime, compliance, risk, and AI experts helps train and evaluate next-generation AI models through expert AML knowledge, transaction monitoring expertise, and regulatory compliance experience.

Role Overview

We are looking for an AI/ML Engineer to work primarily on managing and evaluating existing AI/ML systems, analyzing AI agents, identifying model failures, and improving overall system performance.

The role requires a strong understanding of LLM/AI fundamentals, model evaluation, AI agent architecture, RAG, model improvement/fine-tuning, and production ML systems.

This is a hands-on role for someone who can not only evaluate AI systems but also understand why models or agents fail and identify practical ways to improve their performance.

Key Responsibilities:

• 3+ years of experience in AI/ML, Manage and monitor existing AI/ML and GenAI systems.
• Conduct structured LLM and model evaluations using defined benchmarks and evaluation criteria.
• Analyze AI/agent outputs and identify accuracy, reasoning, retrieval, tool-use, and behavioral failures.
• Analyze AI agent architecture and workflows to identify failure points and performance bottlenecks.
• Evaluate and improve RAG pipelines, including retrieval quality and response generation.
• Identify opportunities for model improvement, prompt optimization, and fine-tuning.
• Analyze evaluation results and translate findings into actionable improvements.
• Support testing and validation of new model/agent versions.
• Work with production ML systems and help identify reliability, latency, scalability, and performance issues.
• Collaborate with engineering and AI teams to implement and validate improvements.
• Document evaluation results, failure patterns, root causes, and recommended solutions.

Must-Have Skills:

• Strong understanding of LLM and AI fundamentals.
• Experience working with production ML/AI systems.
• Hands-on experience with model and LLM evaluation.
• Understanding of AI agent architecture, agentic workflows, and failure analysis.
• Experience with RAG (Retrieval-Augmented Generation) systems.
• Understanding of model improvement, fine-tuning, and prompt optimization.
• Strong Python programming and analytical skills.
• Ability to investigate model failures and perform root-cause analysis.
• Ability to work with evaluation datasets, benchmarks, metrics, and structured testing.

Who Would Be a Good Fit

We're looking for someone who has worked beyond simply building AI models and has hands-on experience with evaluating, debugging, analyzing, and improving AI/LLM systems.

Candidates with experience in LLM evaluation + AI agents + RAG + model improvement + production ML systems would be particularly relevant..

Why Join Us

• Work on real-world LLM and agentic AI systems.
• Gain hands-on exposure to AI evaluation and model improvement.
• Work on challenging problems involving AI agent failures, RAG, and model performance.
• Opportunity to contribute to production-oriented AI projects.