Role Summary
We are hiring a QA Engineer to own end-to-end quality for our data-heavy and AI-powered applications. You will be the first line of defense against bad numbers, hallucinated answers, biased predictions, and silent data drift — the kinds of bugs that never show up as error messages but quietly erode user trust.
You will work closely with data engineers, ML engineers, product managers, and domain experts to define what correct means, design test strategies, automate regression coverage, and ship with confidence. This is not a click-through role; it is a numerical, analytical, and investigative one.
Key Responsibilities
- Own test strategy for data analytics dashboards, ML models, and Gen AI features from requirement to release.
- Validate calculations, aggregations, and KPIs by independently recomputing numbers in SQL, spreadsheets, or Python.
- Design evaluation frameworks for LLM-powered features, including hallucination checks, prompt regression, safety guardrails, and consistency tests.
- Build and maintain automated regression suites for data pipelines, dashboards, APIs, and model outputs — integrated into CI/CD.
- Investigate and document data discrepancies, model drift, and output anomalies; work with engineering to root-cause and resolve.
- Contribute to the definition of metric contracts and data quality rules so that every published number has a single agreed definition.
- Write clear, reproducible bug reports with SQL, screenshots, and expected-versus-actual values.
- Review test coverage for new features during design review — catch gaps before code is written.
- Mentor other team members on data and AI quality practices as the team scales.
Required Qualifications
- 3+ years of QA experience, ideally on data-heavy, analytics, or reporting products.
- Strong SQL — able to read, write, and debug queries independently; comfortable with joins, aggregations, window functions, and subqueries.
- Fluent with Excel or Google Sheets for cross-verification of numbers, pivot tables, and formula-based validation.
- Experience with at least one UI automation framework (Selenium, Playwright, Cypress, or equivalent).
- Experience with API testing using Postman, REST Assured, or similar.
- Basic statistical literacy — understands mean, median, percentiles, distributions, and why they differ.
- Strong written communication — can write a bug report that a developer immediately understands and a stakeholder immediately trusts.
- A skeptical mindset — treats every number as a claim that must be verified, not a fact.
Preferred Qualifications
- Python scripting for data validation (pandas, Jupyter notebooks).
- Experience testing ML models — familiarity with evaluation metrics such as accuracy, precision, recall, F1, ROC-AUC.
- Exposure to LLM evaluation — prompt regression, output similarity scoring, hallucination detection, red-teaming.
- Familiarity with vector databases, embeddings, or RAG architectures.
- Hands-on experience with any model evaluation platform (Langsmith, Weights & Biases, MLflow, Evidently, or similar).
- Performance testing experience (JMeter, k6, or Locust).
- Background in healthcare, fintech, insurance, or other regulated data domains.
- CI/CD integration experience (GitHub Actions, Jenkins, CircleCI).
Skills: ml,testing,numbers,analytics,sql,ci,cd,prompt,data
