Overview
Agentic AI Observability Senior Engineer is responsible for deploying, integrating, and operating a scaled Agentic AI observability platform across both internal and external agent frameworks. This role focuses on production-ready instrumentation and telemetry pipelines that provide end-to-end visibility across multi-step agent workflows—including planner/executor loops, tool/function calls, RAG retrieval, and memory/state—ensuring reliability, safety, performance, and cost governance at scale
Responsibilities
- Agentic AI Observability at Scale (0%)
- Platform Deployment & Operations (Agentic AI Observability at Scale)
- Deploy and run the Agentic AI observability platform across dev/uat/prod with HA, resiliency, and controlled rollouts
- Implement release automation (CI/CD), canary deployments, rollback strategies, and configuration management for platform components
- Own operational readiness: on-call runbooks, incident response, and production support for agent observability services
- End-to-End Agent Workflow Tracing (Planner → Tools → Retrieval → Response)
- Implement distributed tracing for full agent execution graphs, including correlation across: prompts, intermediate reasoning steps (where permitted), tool calls, external APIs, retrieval queries, and final responses
- Enforce consistent trace context propagation, correlation IDs, and semantic conventions across agent services
- Build instrumentation patterns to represent agent flows as spans (e.g., plan span, tool span, retrieval span, memory span, response span)
- Agent Framework Integrations & Standardized Instrumentation
- Deploy and maintain integrations for internal agent frameworks and external ecosystems such as Crew.ai, LangChain, Semantic Kernel, AutoGen, and custom orchestrators
- Create reusable SDKs/middleware/sidecar patterns for teams to instrument agents with minimal effort
- Define and implement tagging standards for: agent name/version, tool name, model provider, prompt template ID, retrieval source, tenant/app, and environment
- Agentic AI Telemetry Pipelines & AI-Specific Signals
- Build scalable pipelines for agent telemetry (logs/metrics/traces) using OpenTelemetry and platform observability tooling
- Capture AI-specific metrics including: token usage, cost per task, tool-call latency, retrieval latency, grounding score proxies, error rates, and agent loop iterations
- Implement sampling and redaction strategies for sensitive agent payloads (prompts, responses, retrieved content) aligned to governance requirements
- Collaboration with Teams (10%)
- Collaborate with transformation teams and business stakeholders to understand requirements and tailor AI agents to specific domains.
- Work closely with AI platform teams to build scalable and cross-domain AI agents while ensuring end-to-end observability.
- Integration & Deployment (10%)
- Build and maintain CI/CD pipelines for agent services and operations center components, including automated testing and deployment
- Automate onboarding for new agent use cases (templates, scaffolding, configuration checks)
- Drive best practices for secure, scalable, and cost-effective agent deployments
- Continuous Learning (10%)
- Stay updated with the latest advancements in AI and machine learning technologies and integrate these into existing or new AI agents.
- Conduct thorough testing and validation to ensure the reliability and accuracy of AI agents and solutions.
Qualifications
- Education: Bachelor's or Masters in Computer Science, AI/ML, Data Science, or a related field.
- Experience: 4-8+ years of software engineering experience; 2-3+ years building and observe AI/ML or GenAI applications preferred
- Required Expertise:
- Strong hands-on experience deploying observability solutions (Prometheus/Grafana/Elastic/Splunk/Datadog or equivalent)
- Deep working knowledge of OpenTelemetry instrumentation and telemetry pipeline operations
- Experience observing agentic AI systems: tool/function calls, orchestration, routing, memory/state, and RAG pipelines
- Familiarity with Crew.ai, LangChain, Semantic Kernel, AutoGen, or similar agent frameworks
- Experience with evaluation/quality monitoring and safe logging strategies for LLM systems
- FinOps experience for tracking token and GPU spend, chargeback/showback, and cost anomaly detection
- Experience implementing data governance controls for AI telemetry (PII redaction, retention, auditability)
- Strong Kubernetes experience (AKS/EKS/GKE) including Helm, operators, ingress, and service networking
- Strong automation skills (Python/Bash/Go) and CI/CD experience
- Infrastructure-as-Code (Terraform/Bicep/CloudFormation)
- Agent workflow tracing and telemetry correlation
- Production operations and debugging distributed systems
- Observability-as-a-platform enablement and automation
- Strong documentation, collaboration, and stakeholder influence
- Technical Proficiency: Implement monitoring for agent failure modes: tool-call failures, infinite loops, timeouts, hallucination risk signals, retrieval misses, and degraded response quality. Create alerts aligned to operational SLOs (availability, latency, tool reliability) and AI-specific indicators (cost spikes, loop bursts, retrieval anomalies). Support guardrail observability: policy blocks, content filtering events, and safety classifier outcomes (where applicable). Build onboarding automation (IaC, templates, CI checks) that makes observability default-on for all agentic services.
- Problem-Solving: Ability to translate business challenges into technical solutions.
- Collaboration Skills: Effective at working within cross-functional teams.
- Agility: Flexibility to adapt to changing requirements and new technologies.
- Communication Skills: Capable of explaining complex technical concepts to non-technical stakeholders.
