The Role
The QA Architect, Data Products is a senior individual contributor and technical lead who owns the quality of our data products end-to-end and shapes the testing strategy across application and AI products in partnership with the QA Manager.
What makes this role unusual is the combination it requires: the product intuition to understand what working correctly actually means for a benefits platform customer, the data engineering literacy to test complex pipelines, the AI-native fluency to use generative tools as a daily accelerator, and the hands-on automation depth to implement what you design. You will also mentor a small team of QA engineers - not as a manager, but as the most technically advanced person in the room. If you have spent time thinking about quality from both a product outcome lens and a data systems lens, and you want to help define what AI-native testing looks like for a company that processes healthcare benefits at scale, this role was written for you
How Success Will Be Measured
• Coverage and quality of test strategy artifacts across data, application, and AI products.
• Synthetic data capability: coverage of customer configuration variations, HIPAA compliance, and team adoption.
• Data product quality: reduction in data incidents, schema breaks, and pipeline defects reaching downstream consumers or customers.
• Automation coverage and reliability at system and component layers for your products.
• AI tooling adoption within the team: how many engineers are using AI-assisted workflows you introduced.
• Mentorship outcomes: skill growth of QA engineers you work with, measured through delivery quality and career progression.
What You Bring
- Experience. 7–10 years in software quality engineering with clear progression into technical leadership or architecture. You have designed test strategies, not just executed them.
- Product background. You have spent meaningful time working from a product outcome lens - validating business behavior, partnering with product managers, and thinking about what the customer actually experiences. Prior experience as a QA lead embedded in a product team, or a background that includes product testing, QA ownership of a product area, or time working alongside PMs on UAT.
- Data product testing. Hands-on experience testing data pipelines, ETL/ELT workflows, data APIs, or reporting products. You understand schema validation, data quality frameworks, and what it means to test data correctness at each stage of a pipeline.
- Synthetic data and test data architecture. Experience designing test data strategies and building or using synthetic data generation - ideally with AI tooling. You understand the constraints that HIPAA and PII place on test data in a healthcare benefits context.
- Configuration-variation testing. Experience testing highly configurable or multi-tenant platforms where customer setups vary significantly. You have built frameworks that handle this systematically.
- Automation depth. Strong hands-on automation skills at system and component levels. You write clean, maintainable test code in at least one major language and framework (Python, Java, TypeScript, Playwright, Pytest, TestNG, or similar).
- AI-native, not AI-adjacent. You are using AI tools actively in your testing practice today: generating test cases, producing synthetic data, using LLMs in exploratory and analytical tasks. You have opinions formed from daily use, not vendor demos.
- AI and ML product testing. Exposure to evaluating AI or ML model outputs: evaluation harnesses, prompt regression, drift or bias detection. You do not need to be an ML engineer, but you need to work productively with them.
- Education. Bachelor's in Computer Science, Engineering, Information Systems, or equivalent practical experience.
Nice to Have
- Experience with benefits administration, HSA/FSA/HRA, payments, or similar multi tenant financial or healthcare SaaS.
- Familiarity with data quality tools such as Great Expectations, dbt tests, Monte Carlo, or similar.
- Hands-on with ML evaluation frameworks such as DeepChecks, Giskard, Ragas (for RAG pipelines), or custom eval harnesses.
- Prior experience in a formal QA Center of Excellence or platform quality role.
- Contributions to open-source testing tools, conference talks, or written work on AI-native testing practices.
