AI/ML Ops Engineer Agentic AI Governance Platform

Company Description Yodaplus is an AI-first technology company focused on transforming enterprises in BFSI, maritime, supply chain, and retail through advanced Agentic AI solutions. Built by practitioners with deep industry experience, Yodaplus delivers AI products that automate complex, judgment-intensive workflows at scale. Its portfolio includes solutions such as GenRPT Finance for automated financial research and reporting, KYC Bharat for end-to-end KYC automation, a Credit Limits Monitoring Framework for intelligent credit decisioning, and OceanDocs AI for maritime document intelligence. Yodaplus is trusted by development banks, investment banks, shipping majors, and retail enterprises across India, the US, UAE, and the UK. The company is dedicated to engineering the next layer of enterprise intelligence rather than just implementing technology.

AI/ML Ops Engineer — Agentic AI Governance Platform

Role Overview

We are building an agentic AI governance platform for regulated financial services. The platform uses AI agents to guide users through onboarding, evidence collection, documentation, gap detection, remediation, approval preparation, and audit reconstruction.

We are looking for an AI/ML Ops Engineer to own the operational backbone of the AI platform: LLM integration, prompt/version management, evaluation, observability, guardrails, embeddings, retrieval quality, monitoring, and deployment reliability.

This role is critical because the platform cannot just call an LLM. It must run AI agents in a controlled, explainable, auditable, and enterprise-safe way.

Responsibilities

The AI/ML Ops Engineer will be responsible for building and operating the AI/ML and agent runtime foundation.

Key responsibilities include:

• Set up and manage LLM integration patterns for enterprise
• Support agent workflows built using LangGraph or similar agent frameworks
• Implement prompt versioning, prompt testing, and structured output validation
• Build evaluation frameworks for agent responses, evidence mapping, gap detection, and documentation quality
• Implement RAG pipelines using embeddings, PostgreSQL, pgvector, and full-text search
• Improve retrieval quality through chunking strategy, metadata design, ranking, reranking, and source attribution
• Build monitoring for LLM calls, tool calls, latency, cost, token usage, errors, hallucination risk, and retrieval quality
• Create automated test sets and golden datasets for lifecycle use cases
• Implement guardrails for unsafe responses, unsupported recommendations, missing citations, and policy violations
• Support audit logging of prompts, responses, retrieved evidence, tool usage, and user approvals
• Help define AgentOps standards for reliability, explainability, rollback, and human-in-the-loop controls
• Work with backend engineers to expose AI services through secure APIs
• Work with product engineers to make AI outputs transparent, reviewable, and source-grounded
• Support deployment, CI/CD, environment management, and production readiness for AI services

Required Skills

• Strong Python engineering experience
• Experience with LLM APIs, embeddings, RAG, and structured outputs
• Experience with MLOps, LLMOps, or AI platform operations
• Experience with prompt management, evaluation, and monitoring
• Experience with PostgreSQL and vector databases such as pgvector, Pinecone, Weaviate, or similar
• Experience building data pipelines for documents, embeddings, metadata, and retrieval
• Experience with Docker, CI/CD, cloud deployment, and environment management
• Strong understanding of testing, observability, logging, and production support
• Ability to build controlled AI systems rather than experimental demos

Preferred Skills

• Experience with LangGraph, LangChain, Semantic Kernel, CrewAI, AutoGen, or similar frameworks
• Experience with MLflow, LangSmith, Weights & Biases, Arize, TruLens, Ragas, OpenTelemetry, or similar tools
• Experience with governance,  compliance, audit, or financial services
• Experience with human-in-the-loop AI systems
• Experience with document intelligence, evidence extraction, and source-grounded generation
• Experience with policy-as-code, guardrails, or AI safety evaluation
• Experience with cloud platforms such as AWS, Azure, or GCP

Technical Stack

Area     Expected Technology

Language        Python

Agent Framework     LangGraph preferred

LLM      Enterprise-approved LLM

Embeddings Enterprise- Approved embedding

Backend          FastAPI

Database       PostgreSQL

Vector Search            pgvector

Search              PostgreSQL full-text search

Evaluation      Ragas / TruLens / custom evals / LangSmith-style tracing

Observability               OpenTelemetry, logs, metrics, traces

Deployment  Docker, CI/CD, cloud-native deployment

Testing              Pytest, evaluation datasets, regression test suites