AI Model Validation Project Manager
Years of Experience: 8-10 Years
Job Summary: We are seeking an experienced AI Model Validation Project Manager to establish and oversee a comprehensive validation framework for AI models. This role will involve curating test datasets, integrating risk-based scoring frameworks, and ensuring robust stakeholder management. The ideal candidate will have a strong background in AI and machine learning, with a focus on model evaluation and validation processes.
Responsibilities:
- AI Model Validation Technical Assessment:
- Establish a high-level standardized validation framework to assess accuracy, fairness, robustness, and reliability across various AI model types and use cases.
- Curate test and validation datasets to ensure comprehensive testing of AI models.
- Integrate risk-based scoring frameworks to classify models by impact and sensitivity, guiding the depth of validation required.
- Design tailored testing frameworks for individual AI models based on their risk profiles.
- Expand validation coverage to include adversarial testing, edge-case analysis, challenger models, and stress testing for high-risk deployments.
- Automated and Streamlined Model Evaluation:
- Develop a library of reusable tests for similar use cases, such as standard tests for OCR and cross-sell models.
- Embed validation checkpoints into CI/CD pipelines and the Software Development Life Cycle (SDLC) for continuous oversight of model performance.
- Create modular evaluation tools for Traditional AI, Generative AI, and Agentic AI systems to ensure flexibility and reuse.
- Implement processes for detecting model degradation, including drift detection, and establish alerting mechanisms to monitor model behavior post-deployment.
- Stakeholder Management:
- Collaborate with stakeholders across the organization, including data scientists, engineers, business leaders, and LBUs, to ensure clear requirements and buy-in for decisions.
- Fluently discuss technical and business concepts with stakeholders to facilitate understanding and collaboration.
- Lead initiatives and gain commitment to change through effective influence.
- Establish and maintain relationships with external vendors.
- Other Responsibilities:
- Stay updated on industry trends and technical developments within the AI ecosystem.
- Participate in industry forums and working groups to advance AI adoption.
- Remain aligned with Prudentials AI Objectives and AI Strategy.
Mandatory Skills:
- Proven experience in developing and deploying AI and machine learning solutions in production environments.
- Strong understanding of AI and machine learning evaluation methods, including examples such as precision, recall, F1 score, and ROC-AUC.
- Excellent Communication Skills, with a demonstrated ability to build relationships across diverse teams.
- Strong experience with Azure DataBricks.
- Experience with WEB API development.
- Ability to thrive in a dynamic and fast-paced global organization, characterized by rapid changes and evolving priorities.
- Data Modeling expertise is a deal-breaker for this role.
Preferred Skills:
- Familiarity with Google Cloud (VertexAI) is a plus.
- Experience with deployment, DataOps, or MLOps, with a preference for ML or AI Ops.
Qualifications:
- Bachelor's degree in Computer Science, Data Science, or a related field; advanced degree preferred.
- 8-10 years of relevant experience in AI model validation, evaluation, and project management.
