AI Reliability Operations Engineer

OPPORTUNITY OVERVIEW

An external threat mitigation cybersecurity company protects organizations from external threats across the public, deep, and dark web. Now we're redesigning how this external threat mitigation cybersecurity company itself works, using AI, and we're doing it alongside the executives who run the company. Teams across the external threat mitigation cybersecurity company already run AI agents and workflows as part of their daily work.

As an AI Reliability Operations Engineer, you own how well those AI solutions work, and you make them better every week, not just keep them running. You run the evaluations that show whether an agent still does its job after a change, find failures early, and turn each one into a test so it stays fixed. You also give business teams the tools to run and tune their own solutions, and you keep the shared AI platforms underneath them healthy.

You're someone who:

• Fixes things before being asked. When you see a weak result or a slow process, you improve it.
• Uses AI on everything, including your own work: evals, scripts, runbooks, and incident notes.
• Treats each solution you look after like your own product: you know how it performs and you push to make it better.

You'll work closely with our AI engineers in the US and with the business teams who use what we build. This role is for someone who wants to get very good at making AI systems reliable, and to see that work show up in how the company runs.

Role and responsibilities

• Own evaluation for every agent, prompt, and model change: run the suites and report results against agreed quality standards.
• Improve the test datasets every week, turning each real failure into a new test case.
• Find regressions, back them with clear evidence, and drive each one to a fix with the right people.
• Keep business teams AI solutions working: catch drift in output, and fix failed runs, broken integrations, and expired access.
• Build tools that let business teams run and adjust their own solutions, such as dashboards, alerts, and simple ways to change settings.
• Fix breakage yourself, and own changes to how a solution behaves, working closely with the department that uses it and the US team.
• Hold every solution you take on to a complete handoff pack: a runbook, evals, and a named department owner.
• Own the runbooks for every solution and platform you work on, and keep them current.
• Own the health of the shared AI platforms: respond to alerts and incidents, and pull in the right people quickly when an issue needs them.
• Own deploys, environment setup, access requests, and configuration changes, following the approved change process.
• Keep tracing, logging, and dashboards current, so anyone can see what an agent did in production.
• Track and report platform cost and usage.
• Automate repeat operational tasks, using AI wherever it speeds up the work.

Required qualifications and skills

• 3+ years in production support, QA or test engineering, data-quality work, or cloud operations.
• Hands-on experience with large language models (LLMs) and how their output varies.
• Experience with testing, evaluation, or data-quality work, including building or running test sets.
• Working skill in Python or TypeScript.
• Experience with a major cloud platform (AWS preferred), including identity and access, CI/CD (continuous integration and delivery pipelines), and monitoring.
• Calm incident handling and clear written escalation.
• Clear written English for runbooks, incident notes, and change notes.
• Good judgment about access and data sensitivity.
• Evidence of initiative and AI fluency: tools, tests, or automations you built without being asked, and daily hands-on use of AI tools in your own work.

Desired qualifications and skills

• Experience with LLM evaluation frameworks and LLM-as-judge scoring (using one model to grade another model's output).
• Experience with managed agent runtimes on AWS, such as Amazon Bedrock AgentCore.
• Experience with observability and tracing tools for LLM or agent systems.
• Experience with modern web hosting and deployment platforms, such as Vercel.
• Experience with infrastructure-as-code.
• Experience supporting internal business users, or on a support or on-call rotation.
• Experience at a security company, or with security-sensitive systems.