Principal Software Engineer
Cadence- Posted an hour ago
- Be among the first 10 applicants
Job Description
At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology.
Senior AI Test / Automation Engineer
Overview
Role: AI Test / Automation Engineer
Location: Bangalore India
Department: AI Engineering / Quality Assurance
Experience Level: Mid to Senior
We are looking for a highly motivated SeniorAI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.
Key Responsibilities
- Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
- Develop comprehensive test suites, including:
- Unit, integration, and end-to-end (E2E)
- Functional, regression, performance, and safety testing
- Validate AI system behavior, including:
- Non-deterministic LLM outputs
- Hallucinations and edge cases
- Multi-step agent decision-making
- Design and manage evaluation systems:
- Golden datasets
- Benchmarking pipelines (accuracy, latency, reliability)
- Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
- Implement observability and telemetry to enable traceability, monitoring, and audit readiness
- Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
- Track and report quality KPIs, including test coverage, defect leakage, and system reliability
- Drive root-cause analysis and continuous improvement across the AI testing lifecycle
Required Skills
Core Engineering
- Strong programming skills in Python familiarity with Bash, TypeScript, or Go
- Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
- Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
- Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)
AI / ML & Agentic Systems
- Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
- Familiarity with:
- RAG architectures and vector databases (Pinecone, Weaviate)
- Agent frameworks (LangChain, LlamaIndex, AutoGen)
AI Testing Techniques
- Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
- Knowledge of evaluation methods:
- LLM-as-a-judge
- BLEU, ROUGE, semantic similarity scoring
- Experience with prompt and agent regression testing
- Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection
Tooling (Preferred)
- AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
- Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
- Monitoring: Prometheus, Grafana, OpenTelemetry
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and cross-functional collaboration
- Data-driven mindset with focus on quality KPIs
- Detail-oriented with a strong bias toward automation and scalability
Experience Requirements
- 7+ years in QA, SDET, or test automation engineering
- Proven experience building and scaling automation frameworks
- Hands-on experience with AI/ML systems or LLM-based applications
- Experience testing RAG pipelines or agentic workflows
- Owned end-to-end AI test strategy and architecture
- Defined quality metrics and release gates
- Delivered scalable validation pipelines for production AI systems
- Supported audit and compliance readiness
Preferred
- Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
- Exposure to:
- Shift-left testing practices
- Production observability and monitoring
- Chaos or resilience testing
Senior-Level Differentiators
Education
- Bachelor's or Master's degree in Computer Science, Software Engineering, or related field
Nice-to-have:
- ISTQB certification
- Cloud/ML certifications (AWS, Azure, GCP)
- AI testing certifications
What Success Looks Like
- AI systems that are accurate, reliable, and safe
- Fully automated test pipelines integrated into CI/CD
- Measurable improvements in defect leakage and model quality
- Strong observability and auditability across AI systems
- Scalable validation frameworks supporting rapid AI innovation
We're doing work that matters. Help us solve what others can't.
More Info
About Company
Additional Jobs Equal Employment Opportunity Policy: Cadence is committed to equal employment opportunity throughout all levels of the organization. Read the policy(opens in a new tab) We welcome your interest in the company and want to make sure our job site is accessible to all. If you experience difficulty using this site or to request a reasonable accommodation, please contact staffing@cadence.com . Privacy Policy: Job ApplicantIf you are a job seeker creating a profile using our careers website, please see the privacy policy(opens in a new tab) . E-VerifyCadence participates in the E-Verify program in certain U.S. locations as required by law. Download More Information on E-Verify(64K) Cadence plays a critical role in creating the technologies that modern life depends on. We are a global electronic design automation company, providing software, hardware, and intellectual property to design advanced semiconductor chips that enable our customers create revolutionary products and experiences. Thanks to the outstanding caliber of the Cadence team and the empowering culture that we have cultivated for over 25 years, Cadence continues to be recognized by Fortune Magazine as one of the 100 Best Companies to Work For. Our shared passion for solving the world's toughest technical challenges, our dedication to pushing the limits of the industry, and our drive to do meaningful work differentiates the people of Cadence. Cadence is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, sex, age, national origin, religion, sexual orientation, gender identity, status as a veteran, basis of disability, or any other protected class.
