ALM - Gen AI Quality Engineering
Required Skills & Qualifications
- 8-12 years of experience in Quality Engineering
- 3+ years of hands-on experience in testing GenAI applications/
- Strong expertise in API testing, automation frameworks and release governance
- Deep understanding of:
- RAG
- Prompt engineering
- LLM evaluation techniques (Evals)
- Agent workflows
- Strong knowledge of API security testing and OAuth-based systems
- Experience integrating testing within CI/CD pipelines
- Cloud platform familiarity (AWS)
- Excellent communication and stakeholder management skills
Role Summary
We are seeking a Senior Lead Quality Engineer (Gen AI) to define and drive the end-to-end Quality Engineering strategy for AI-powered systems, including LLM-based applications, AI assistants, Retrieval-Augmented Generation (RAG) pipelines, and agent-driven workflows.
This role will establish measurable AI quality standards, governance frameworks, automation strategy, and release readiness criteria. The ideal candidate brings strong QE leadership along with hands-on expertise in LLM behavior, contextualization techniques (Prompt Engineering, RAG, Fine-Tuning), evaluation frameworks (Evals), accessibility, API security, and AI observability.
You will work cross-functionally with Product, Engineering, and Platform teams to ensure AI systems are accurate, safe, reliable, scalable, and production-ready.
Key Responsibilities
AI Quality Strategy
- Define and own the AI Quality Engineering roadmap and standards
- Establish measurable success criteria for Gen AI features
- Design quality gates and release readiness frameworks
- Lead AI risk assessment and regression strategy
AI Feature & Evaluation Understanding
Define and track AI-specific KPIs, including:
- Intent accuracy
- Context retention (multi-turn reliability)
- Hallucination rate
- Bias & toxicity scores
- Guardrail breach frequency
- Fallback & recovery rate
- Retrieval precision/recall (RAG)
- Token efficiency
- P95/P99 latency
Additional responsibilities:
- Design benchmark datasets for deterministic AI regression
- Implement automated and human-in-the-loop evaluation frameworks
- Build structured AI scoring rubrics and quality dashboards
Contextualization & LLM Validation
- Design and validate prompt engineering strategies
- Validate RAG (chunking, embeddings, retrieval relevance)
- Evaluate fine-tuned models and dataset integrity
- Validate agent-based and multi-step reasoning workflows
Safety, Security & Responsible AI
- Ensure PII protection and data boundary enforcement
- Conduct API security validation, including:
- Authentication & OAuth flows
- Role-based access control
- Cross-account/user isolation
- Rate limiting and throttling
- Malformed input and abuse scenarios
- Ensure compliance with enterprise AI governance standards
Accessibility
- Validate compliance with WCAG standards
- Ensure screen reader and keyboard navigation compatibility
Performance, Reliability & Observability
- Define and validate latency benchmarks and SLAs
- Conduct scalability and concurrency testing
- Monitor token usage and cost efficiency
- Validate feedback, fallback mechanisms and recovery behavior
Automation & CI/CD Integration
- Build AI-focused automated regression frameworks
- Integrate AI validation into CI/CD pipelines
- Enable continuous AI quality monitoring across releases
- Ensure backward compatibility across API and platform versions
About Adobe
Adobe empowers everyone to create through innovative platforms and tools that unleash creativity, productivity and personalized customer experiences. Adobe's industry-leading offerings including Adobe Acrobat Studio, Adobe Express, Adobe Firefly, Creative Cloud, Adobe Experience Platform, Adobe Experience Manager, and GenStudio enable people and businesses to turn ideas into impact, powered by AI and driven by human ingenuity.
Our 30,000+ employees worldwide are creating the future and raising the bar as we drive the next decade of growth. We're on a mission to hire the very best and believe in creating a company culture where all employees are empowered to make an impact. At Adobe, we believe that great ideas can come from anywhere in the organization. The next big idea could be yours.
Let's Adobe together
At Adobe, we believe in creating a company culture where all employees are empowered to make an impact. Learn more about Adobe life, including our , , , comprehensive , the , the we serve, and how you can help us advance our mission of empowering everyone to create.
Adobe is proud to be an employer. We do not discriminate based on gender, race or color, ethnicity or national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, or any other protected characteristic.
Adobe aims to make our Careers website and recruiting process accessible to any and all users. If you have a disability or special need that requires accommodation to navigate our website or complete the application process, email or call +1 408-536-3015.
AI Use Guidelines for Interviews:
Our interviews are designed to reflect your own skills and thinking. The use of AI or recording tools during live interviews is not permitted unless explicitly invited by the interviewer or approved in advance as part of a reasonable accommodation. If these tools are used inappropriately or in a way that misrepresents your work, your application may not move forward in the process.
At Adobe, we empower employees to innovate with AI - and we look for candidates eager to do the same. As part of the hiring experience, we provide clear guidance on where AI is encouraged during the process and where it's restricted during live interviews. See how we think about .
