We are a global healthcare leader headquartered in Indianapolis, Indiana. Our 42,000+ employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We're looking for people who are determined to make life better for people around the world. Come build advanced software capabilities to accelerate our digital transformation and support Lilly's evolution to be the leader in Pharma-tech! THE ROLE The Software Product Engineering (SPE) organization is actively looking for a motivated Principal Engineer u2013 QE (R3) who is deeply technical and passionate about engineering quality at scale. This is a core individual contributor role for a hands-on quality engineer with deep expertise in building evaluation frameworks for AI platforms u2014 including chat and voice bot applications u2014 alongside robust test automation frameworks for API, web, and mobile testing. Are you a quality-engineering craftsperson who thrives on solving complex technical problems Do you have a passion for AI quality validation, framework architecture, and continuous innovation Do you want to shape Lilly's engineering direction as a leader in Pharma-tech If so, please apply. WHAT YOU'LL BE DOING In this role, you will be at the technical forefront of quality engineering u2014 designing and building evaluation frameworks for AI-based platforms including LLM-powered chat and voice bots, and developing scalable automation frameworks for API, web, and mobile applications. You will work hands-on with Python, Java, and JavaScript/TypeScript to build production-grade test infrastructure, integrate with CI/CD pipelines, and apply advanced AI testing methodologies. You will also contribute to performance testing, cloud-based test execution, and the continuous improvement of quality practices across a global cross-functional team. KEY RESPONSIBILITIES AI Platform & Evaluation Framework Engineering u25AAu00A0 Designs and builds end-to-end evaluation frameworks for AI platforms including LLM-based chat bots and voice bots , covering accuracy, relevance, hallucination detection, latency, and response quality. u25AAu00A0 Implements AI model evaluation pipelines using RAGAS, DeepEval, and LangChain to benchmark and validate LLM outputs against ground truth. u25AAu00A0 Develops test strategies for conversational AI u2014 validating intent recognition, slot filling, dialogue flow, fallback handling, and multi-turn context retention. u25AAu00A0 Builds voice bot quality validation covering speech recognition accuracy, TTS quality, call flow logic, and DTMF handling. u25AAu00A0 Defines quality metrics for AI systems including precision, recall, F1-score, BLEU/ROUGE scores, and semantic similarity thresholds. u25AAu00A0 Collaborates with AI/ML engineers and data scientists to integrate evaluation frameworks into model training and deployment pipelines. Test Automation Framework Development u25AAu00A0 Architects and builds robust, scalable, and maintainable test automation frameworks for API (REST/SOAP), web, and mobile applications using pytest, Selenium, WebdriverIO, and Appium. u25AAu00A0 Designs framework components including page object models, data-driven layers, reporting modules, parallel execution engines, and retry mechanisms. u25AAu00A0 Develops API automation suites covering contract testing, schema validation, authentication flows, and end-to-end service integration testing. u25AAu00A0 Implements mobile test automation for both iOS and Android platforms, covering native, hybrid, and responsive web applications. u25AAu00A0 Builds reusable test libraries and utility modules in Python, Java, and JavaScript/TypeScript to accelerate framework adoption across teams. u25AAu00A0 Integrates test frameworks with CI/CD pipelines using GitHub Actions, Jenkins, or Bamboo for continuous test execution and reporting. Technology Leadership & Engineering Excellence u25AAu00A0 Serves as a key technical contributor and subject matter expert in quality engineering technologies, automation architectures, and testing methodologies. u25AAu00A0 Demonstrates deep understanding of software design patterns, SOLID principles, and architectural best practices applied to test automation. u25AAu00A0 Proactively adopts and evaluates new programming languages and tools including Go, Rust, JavaScript, Python, and TypeScript to improve automation quality and coverage. u25AAu00A0 Develops and maintains CI/CD pipeline integrations, ensuring test suites are reliable, fast, and part of every deployment gate. Performance, Cloud & Infrastructure Testing u25AAu00A0 Designs and executes performance, load, and stress testing strategies using JMeter to validate system reliability, throughput, and scalability under realistic and peak conditions. u25AAu00A0 Drives cloud-based test execution strategies leveraging AWS, Kubernetes, and Docker for dynamic, scalable, and containerised test environments. u25AAu00A0 Implements infrastructure-as-code test provisioning to enable repeatable and version-controlled test environment setup. u25AAu00A0 Conducts API security and reliability testing, including authentication, rate limiting, error handling, and timeout validation. Process Improvement & Quality Advocacy u25AAu00A0 Champions shift-left testing practices, embedding quality validation early in the development lifecycle and reducing defect escape rates. u25AAu00A0 Identifies and implements test optimisation techniques u2014 including parallel execution, smart test selection, and coverage prioritisation u2014 to reduce pipeline duration without sacrificing quality. u25AAu00A0 Implements best practices in test design, maintenance, and documentation, ensuring automation assets remain production-grade over time. u25AAu00A0 Advocates for quality-first engineering across development and product teams, providing data-driven insights on defect trends, test coverage, and automation ROI. Mentorship & Knowledge Sharing u25AAu00A0 Supports and mentors junior and mid-level QA engineers, providing technical guidance on automation frameworks, AI evaluation, and CI/CD integration. u25AAu00A0 Conducts framework and code reviews, ensuring best practices and engineering standards are consistently applied. u25AAu00A0 Shares knowledge through internal talks, documentation, and community-of-practice contributions, advancing the QE capability across SPE. u25AAu00A0 Encourages a culture of continuous learning, staying current with industry trends in AI testing, automation tooling, and software quality. Innovation & Thought Leadership u25AAu00A0 Researches and evaluates emerging testing technologies including AI-based test generation, self-healing automation, and autonomous testing agents. u25AAu00A0 Promotes innovative approaches to quality engineering, driving adoption of AI-driven test automation and intelligent defect prediction within the team. u25AAu00A0 Contributes to the SPE QE community of practice u2014 publishing patterns, reusable components, and technical guidance for cross-team benefit. Agile & DevOps Collaboration u25AAu00A0 Ensures seamless test automation integration within CI/CD pipelines, supporting continuous delivery and deployment across all supported environments. u25AAu00A0 Participates actively in Agile ceremonies including sprint planning, backlog refinement, and defect triaging, contributing technical quality perspectives. u25AAu00A0 Works closely with development, DevOps, and product teams to embed testing within Agile workflows and ensure alignment on release readiness. REQUIRED TECHNICAL SKILLS & QUALIFICATIONS u25AAu00A0 Bachelor's or Master's degree in Computer Science, Engineering, or a related field. u25AAu00A0 10+ years of Quality Engineering experience in a high-tech, life sciences, or related field. u25AAu00A0 Hands-on experience building evaluation frameworks for AI platforms including LLM-based chat bots and voice bot applications. u25AAu00A0 Proficiency in Python, Java, and JavaScript/TypeScript with the ability to build production-grade automation code, not just scripts. u25AAu00A0 Deep expertise in test automation framework design and development for API, web, and mobile using pytest / Selenium / WebdriverIO / Appium. u25AAu00A0 Strong experience in AI/ML testing methodologies including LLM evaluation metrics, hallucination detection, and model benchmarking. u25AAu00A0 Hands-on experience with RAGAS, DeepEval, or equivalent model evaluation frameworks. u25AAu00A0 Strong experience in test strategy formulation, test planning, and risk-based testing approaches. u25AAu00A0 Proficiency in performance and load testing using JMeter or equivalent tools. u25AAu00A0 Strong understanding of cloud computing and containerised testing using AWS, Kubernetes, and Docker. u25AAu00A0 Experience with CI/CD tools (GitHub Actions, Jenkins, Bamboo) and integration of automated tests into delivery pipelines. u25AAu00A0 Experience with GitHub, JIRA, Xray, and Agile delivery methodologies. u25AAu00A0 Excellent written and verbal communication skills, with the ability to convey complex technical concepts to both technical and non-technical audiences. PREFERRED QUALIFICATIONS u25AAu00A0 Experience with LangChain, LangGraph, or similar LLM orchestration frameworks. u25AAu00A0 Hands-on experience testing multi-modal AI applications (text, voice, and image inputs). u25AAu00A0 Knowledge of speech recognition quality metrics (WER, CER) for voice bot evaluation. u25AAu00A0 Experience in multi-region software testing and global test infrastructure. u25AAu00A0 Knowledge of security and compliance testing in regulated environments (e.g., GxP, HIPAA). u25AAu00A0 Contributions to open-source test automation frameworks or AI evaluation tooling.
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form () for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lillyu00A0does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.
#WeAreLilly
