AI Model & Dataset Evaluation
- Define and own evaluation frameworks for ML, Computer Vision, LLM, RAG, and Agentic AI systems.
- Establish statistically rigorous model validation methodologies.
- Assess dataset quality, provenance, coverage, and labeling integrity.
- Conduct significance testing before approving model improvements.
AI Security & Red Teaming
- Lead adversarial testing and red-team exercises.
- Evaluate resilience against data poisoning, evasion attacks, prompt injection, jailbreaks, and tool abuse.
- Develop security benchmarks and release-gating criteria for AI deployments.
- Manage vulnerability identification and remediation tracking.
AIOps & Production Reliability
- Design drift monitoring and operational observability frameworks.
- Build shadow deployment and canary rollout strategies.
- Architect high-throughput, low-latency AI inference pipelines.
- Develop confidence calibration and model reliability monitoring.
Governance & Compliance
- Ensure reproducibility, auditability, and certification readiness.
- Maintain AI lifecycle documentation and evaluation records.
- Enforce secure data handling and access-control practices.
Technical Leadership
- Mentor ML engineers, security researchers, and AI practitioners.
- Establish engineering standards for secure AI development.
- Advise leadership on AI risk, deployment readiness, and technology selection.
Required Skills
- 6 - 10 years of experience in Machine Learning, AI Engineering, Security Engineering, or related domains.
- Strong expertise in:
- Statistical testing and model validation
- Classification and detection evaluation
- Adversarial Machine Learning
- LLM, RAG, and Agentic AI evaluation
- MLOps and production AI systems
- Drift detection and monitoring
- Secure deployment practices in high-security environments
