KEY ACCOUNTABILITIES

Data Quality Policy & Framework Implementation
Define and operationalize enterprise Data Quality policies, procedures, and standards.
Establish standardized data quality dimensions and certification frameworks.
Implement scalable validation frameworks across ingestion, transformation, and serving layers.
Embed quality-by-design principles into data product lifecycle.
Data Observability Platform Development
Design and implement end-to-end data observability capabilities including:
- Data freshness and SLA monitoring
- Volume and distribution anomaly detection
- Schema drift and pipeline health monitoring
- Data lineage validation and reliability tracking
- Develop automated alerting and incident detection mechanisms.
Custom Data Applications (DataApps) Development
Build custom Data Quality and Observability applications using:
- Databricks native capabilities
- Streamlit / Databricks Apps
- Python-based backend services
Develop user interfaces enabling:
- Data quality rule configuration
- Dataset certification workflows
- Quality score visualization
- Issue tracking and remediation workflows
- Enable self-service quality monitoring for engineering and analytics teams.
Azure & Databricks Platform Integration
Implement data quality checks within Azure-based data pipelines and Databricks workflows.
Integrate monitoring with:
- ADLS Gen2
- Databricks Lakehouse architecture
- Batch and streaming pipelines
- Develop reusable frameworks leveraging Spark and Delta Lake.
- Optimize performance and scalability of quality validation workloads.
Automation & Engineering Excellence
Integrate DQ checks into CI/CD and deployment pipelines.
Develop metadata-driven quality monitoring solutions.
Implement automated remediation and self-healing workflows where applicable.
Ensure auditability, traceability, and governance compliance.
Metrics, Reporting & Adoption
Define enterprise Data Quality KPIs and reliability SLAs.
Build dashboards tracking platform-wide data trust scores.
Drive adoption of standardized DQ practices across engineering teams.
Support audit and compliance reporting initiatives.
Data Quality Score
Leadership & Collaboration
Act as technical lead for Data Quality and Observability engineering.
Mentor engineers on best practices for data reliability.
Collaborate with Data Engineering, Governance, and Platform Architecture teams.
Contribute to long-term evolution of the enterprise data platform.

QUALIFICATIONS, EXPERIENCE AND SKILLS

Education

Bachelor's or master's degree in computer science, Data Engineering, Information Systems, or related field.

Experience

8+ years of experience in Data Quality engineering roles within Data Platforms/Data Engineering teams.
Proven experience building custom applications on Databricks or data platforms.
Experience designing enterprise Data Quality or Data Observability solutions.
Hands-on experience developing internal data tools or platform applications.

Technical Skills (Required)

Cloud & Data Platform
Strong expertise in:
Microsoft Azure
Databricks Lakehouse platform
ADLS Gen2
Distributed data processing using Spark
Application & DataApp Development
- Experience building DataApps using:
  - Streamlit
  - Databricks Apps or notebook-based applications
  - Python backend development
  - Experience designing UI-driven data engineering tools or internal platforms.
- Data Quality & Observability
  - Experience implementing data validation frameworks.
  - Strong SQL and Python programming skills.
  - Knowledge of anomaly detection, monitoring, and data reliability concepts.
- Engineering & Integration
  - CI/CD integration for data pipelines.
  - REST API integrations and automation workflows.
  - Metadata-driven architectures and lineage concepts.

Core Competencies

Platform-first engineering mindset.
Strong problem-solving and analytical thinking.
Ability to translate governance requirements into scalable technical solutions.
Strong stakeholder collaboration and communication skills.
Ownership mindset with ability to lead initiatives end-to-end.

Preferred

Experience with Great Expectations, Deequ, Soda, or similar frameworks.
Experience with streaming data validation.
Exposure to AI-driven data observability or anomaly detection.
Experience building enterprise internal developer platforms.

#LI-AA6

Group Lead - Data Quality Engineer