Job Description
We are seeking a Senior AI Platform Engineer to own the quality, safety, and cost governance layer of an enterprise Agentic AI Platform built on Microsoft Foundry. This is a specialist role that spans four interconnected disciplines agent evaluation and QA, model governance, responsible AI enforcement, and AI FinOps. You will build the frameworks, pipelines, and dashboards that ensure every agent deployed to production meets defined standards for accuracy, safety, compliance, and cost efficiency.
The ideal candidate has hands-on experience with LLM evaluation harnesses, prompt management at scale, and cloud cost attribution - with a strong foundation in experimental design and statistical analysis.
Key Responsibilities
Evaluation & QA
Cost Attribution Implement per-agent, per-project, per-department cost tracking using APIM custom dimensions + App Insights + KQL queries. Build Azure Monitor dashboards for token consumption, cost trends, and anomaly detection.
Quota & PTU Management Manage PTU (Provisioned Throughput Unit) allocation across projects. Implement PAYG overflow policies. Track utilization and right-size allocations monthly.
Chargeback Build chargeback/showback reports via Power BI or Azure Monitor Workbooks. Attribute costs to business units. Implement anomaly detection for unexpected token spikes.
Cost Policy Author APIM cost-tag policies. Define and enforce token rate limits per project. Establish monthly FinOps review cadence.
Technical Requirements
Category
Preferred Qualifications
Experience 5+ years in software engineering or ML engineering, with at least 2 years focused on LLM/AI platform quality, e
Certifications Microsoft Certified Azure AI App and Agent Developer Associate (AI-103, replacing AI-102), FinOps Certified Practitioner, or equivalent.
Evaluation Depth Has designed and operated an LLM evaluation harness in production - including defining thresholds, building quality gates, and preventing defective deployments.
FinOps Track Record Has implemented cost attribution for AI/ML workloads with chargeback reporting to business stakeholders.
Responsible AI Hands-on experience configuring content safety filters, PII detection, or prompt injection defenses for production AI systems.
Analytical Mindset Strong foundation in statistical analysis of evaluation results, drift detection, and anomaly identification.
