Principal Engineer – AI Quality & Evaluation Architecture
Role Overview
Vocera (Stryker) is seeking a Principal Engineer to own end-to-end AI quality across the lifecycle — data, models, prompts, evaluation, deployment, and monitoring. This role will define and scale reliable, measurable, production-grade AI systems across speech, NLP, and GenAI in healthcare.
Key Responsibilities
AI Quality Ownership
- Own AI quality across the full lifecycle
- Define SLAs, KPIs, release gates, and production readiness decisions
Evaluation & Reliability
- Build evaluation frameworks for ASR (WER, latency), NLP (intent/entity), and LLMs/RAG (hallucination, safety, groundedness)
- Develop benchmarking, regression pipelines, and golden datasets
- Drive adversarial testing, edge case handling, and failure analysis
AI Testing Platform
- Architect scalable evaluation platforms (offline, regression, A/B, shadow testing)
- Integrate with CI/CD and MLOps pipelines
- Implement monitoring, observability, and drift detection
Data Governance
- Define standards for data curation, annotation, and versioning
- Ensure reproducibility and feedback loops from production
- Maintain healthcare data compliance
MLOps & Continuous Quality
- Establish AI MLOps standards for evaluation, retraining, and deployment
- Enable continuous evaluation and performance monitoring at scale
Leadership
- Act as AI quality authority across the organization
- Mentor teams and align with product and business goals
Qualifications
- 12+ years in software/AI engineering; 5+ years in LLMs, NLP, RAG, or speech
- Experience building scalable AI evaluation frameworks
Expertise in:
- LLM evaluation (hallucination, safety, groundedness)
- Golden datasets, regression testing, adversarial testing
- Prompt validation, Python, data analysis, automation
- CI/CD, MLOps, distributed systems
Nice to Have
- RAG evaluation & retrieval benchmarking
- Speech/ASR evaluation
- Azure ML / OpenAI / AI Search
- Responsible AI & compliance
Stryker is a global leader in medical technologies and, together with its customers, is driven to make healthcare better. The company offers innovative products and services in MedSurg, Neurotechnology, Orthopaedics and Spine that help improve patient and healthcare outcomes. Alongside its customers around the world, Stryker impacts more than 150 million patients annually.
