Key Responsibilities:
- Key Responsibilities
- AI Assurance Architecture
- Architect platforms and frameworks for AI assurance evaluation and benchmarking
- Design systems for LLM agent and RAG evaluation across functional non functional and risk dimensions
- Define architectural patterns for Responsible AI bias detection explainability and safety validation
- Build reusable assurance components supporting Business Assurance Risk Assurance and Reliability
- Security Reliability Governance
- Architect AI testing and validation for security privacy prompt injection and adversarial robustness
- Integrate red teaming threat simulation and chaos style validation for AI systems
- Define governance mechanisms for model usage auditability traceability and compliance
- Ensure AI systems meet enterprise standards for resilience fault tolerance and observability
- Platform Engineering Enablement
- Design AI assurance platforms supporting automated test execution reporting and insights
- Enable integration with CI CD pipelines to enforce AI quality gates
- Collaborate with QE engineering teams to embed AI assurance into the SDLC
- Mentor teams on AI risk identification and mitigation from an engineering perspective
- Core Platforms Frameworks Tooling
- LLM and AI evaluation frameworks PromptFoo DeepEval custom LLM evaluation harnesses
- Prompt RAG and agent validation tooling prompt testing frameworks retrieval accuracy validators agent workflow evaluators
- Responsible AI and model risk tooling Fairlearn SHAP Explainable AI libraries toxicity and bias scanners
- Security and adversarial testing tools for AI systems PyRIT Garak
- AI red teaming and threat simulation frameworks automated red team scripts adversarial test suites for LLMs and agents
- AI assurance automation and QE frameworks Galileo
- Observability for AI behavior and drift Langfuse Arize Evidently custom telemetry dashboards
- Client Orientation Leadership
- Partner with product and engineering teams to identify AI Assurance opportunities and shape roadmaps
- Support client workshops RFPs and solution presentations
- Mentor engineers on AI ML Gen AI best practices and emerging technologies
- Translate complex AI concepts into business friendly narratives
Technical Requirements:
- Must Have Qualifications
- 13 years of experience in software engineering with 3 years in AI with strong architecture ownership
- Hands on expertise in AI ML systems LLM evaluation and assurance frameworks
- Experience with AI red teaming model risk management or AI audit tooling
- Strong understanding of Responsible AI AI risks and governance principles
- Experience with security testing adversarial testing and reliability engineering
- Proficiency in Python automation frameworks and cloud platforms
- Good to Have Skills
- Knowledge of regulatory or compliance considerations for AI systems
- Exposure to performance engineering chaos engineering or resilience testing for AI
- Contributions to internal platforms frameworks or standards
Preferred Skills:
Technology->AI-AI Engineering->LLMOps,Foundational->Project Management->Request For Proposal (RFP)/Proposal Development,Technology->AI-AI Engineering->AI/ML Solution Architecture and Design->traditional ai ml,Technology->Testing Technologyes->Test Automation Technology,Technology->Agile Testing->Agile Testing - ALL->CD/CI,Technology->AI-Generative AI->Conversational AI Platform,Technology->AI-Generative AI->Generative AI - Basic->retrieval augmented generation (rag),Technology->Automated Testing->Automated Testing - ALL
