While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.
If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!
Role: Lead/Associate Lead – QA + MLOps & Generative AI
Experience: 10+ years
Location: Mumbai/Bangalore (Hybrid)
Key Responsibilities
AI/ML & GenAI Testing Strategy (AWS Ecosystem)
Define testing approaches for AI systems built on AWS services such as:
- Amazon SageMaker
- Amazon Bedrock
- AWS Lambda
- Amazon API Gateway
- Amazon Kinesis
- AWS Glue
- Amazon S3
- Amazon CloudWatch
Design Validation Frameworks Covering
- Model accuracy & performance validation
- Data drift & concept drift detection
- Hallucination detection for LLMs
- Prompt robustness testing
- RAG validation (retrieval accuracy + grounding)
- Bias & fairness validation
- Safety & toxicity testing
MLOps Quality Engineering (AWS-Centric)
Validate The End-to-end ML Lifecycle Including
- Data ingestion & feature pipelines
- Model training & hyperparameter tuning
- Model versioning & registry
- Deployment validation
- Canary & blue/green release validation
Work With AWS-native Services Such As
- SageMaker Pipelines
- SageMaker Model Monitor
- SageMaker Feature Store
- Bedrock model evaluation workflows
- CloudWatch-based observability
Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools.
GenAI & Agentic AI Testing
Define Quality Engineering Approaches For
- LLM-based applications using Amazon Bedrock
- Prompt engineering validation
- Multi-agent orchestration testing
- Chatbot & Voice bot conversational testing
- Intent classification validation
- Conversation drift & fallback validation
- API contract validation for LLM integrations
Build Reusable Evaluation Harnesses For
- BLEU / ROUGE scoring
- Embedding similarity scoring
- Response consistency
- Safety scoring frameworks
Framework & Capability Development
- Design reusable AI testing accelerators
- Create AWS-aligned AI test automation frameworks (Python-first)
- Develop synthetic data generation strategies
- Establish AI quality scorecards
- Build an internal AI QA Center of Excellence
Client Engagement & Leadership
- Lead AI/ML quality strategy workshops
- Perform AI risk & readiness assessments
- Present quality architecture to CXOs
- Drive QA transformation programs
- Mentor QA teams on AWS-based AI testing
- Own delivery for AI testing engagements end-to-end
Must Have Skills
Testing Expertise
- 8–12+ years in Quality Engineering
- Strong test strategy, automation & governance experience
- Experience leading QA transformation initiatives
- Experience building frameworks from scratch AI/ML & GenAI Expertise
- Deep understanding of ML lifecycle
- Experience testing ML models (NLP preferred)
- Hands-on experience validating LLM applications
- Strong understanding of:
- Prompt engineering
- RAG architecture
- Embeddings
- Bias & explainability AWS AI/ML Expertise
- Hands-on experience with:
- Amazon SageMaker (training, deployment, monitoring)
- Amazon Bedrock (LLM integration & evaluation)
- S3-based data pipelines
- AWS IAM (security validation)
- CloudWatch monitoring
- Lambda & API Gateway integrations
- AWS CI/CD (CodePipeline / CodeBuild preferred)
Understanding Of
- Infrastructure as Code (Terraform / CloudFormation)
- Observability in AI systems
- Cost monitoring for ML workloads
Technical Skills
- Python (mandatory)
- Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)
- Experience with LLM frameworks (LangChain, etc.)
- API & automation testing frameworks
- Git-based workflows
- Leadership & Communication
- Strong client-facing communication
- Experience leading QA teams
- Ability to create strategy decks & solution proposals
- Strong stakeholder management
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!
