6 Data Quality Automation Analyst (2) (Databricks-Centric)
Role Overview
Design and automate data validation within Databricks pipelines to ensure accurate migration and ongoing data reliability.
Key Responsibilities
- Build automated data quality checks in PySpark
- Develop reconciliation framework between:
- Legacy SQL outputs
- New Databricks outputs
- Implement rule-based validation for healthcare datasets
- Automate profiling and anomaly detection
- Integrate quality checks into pipeline workflows
- Build DQ scorecards and dashboards
- Support cutover validation
Required Skills
- Strong SQL & PySpark
- Experience with Databricks validation frameworks
- Experience implementing automated DQ checks
- Strong reconciliation expertise
- Healthcare data validation experience preferred
