Role Overview
We are looking for an experienced
Senior Data Quality Engineer with strong expertise in
SQL, Databricks, Data Quality, and Data Observability. The candidate will be responsible for implementing data quality controls, monitoring data pipelines, identifying data issues, and ensuring the reliability and accuracy of enterprise data platforms.
Key Responsibilities
- Develop and execute data quality rules, validations, and controls across enterprise datasets.
- Work with Monte Carlo Data Observability to monitor data pipelines, detect anomalies, and identify data reliability issues.
- Develop complex SQL queries for data validation, reconciliation, profiling, and troubleshooting.
- Work hands-on with Azure Databricks for batch and near-real-time data processing.
- Validate ETL/ELT pipelines and identify data quality issues across source-to-target flows.
- Work with relational and NoSQL databases and large-volume datasets.
- Support Data Warehouse, Data Lake, and Lakehouse environments.
- Perform data profiling and investigate data anomalies, freshness, volume, schema, and lineage issues.
- Collaborate with Data Engineers, Architects, Data Stewards, and business teams to resolve data quality issues.
- Support Collibra and data governance initiatives including metadata, lineage, and stewardship.
- Document data quality rules, issues, root-cause analysis, and remediation processes.
- Support production issues and contribute to continuous improvement of data quality and observability processes.
Required Skills
- Strong hands-on SQL experience with large datasets.
- Hands-on experience with Databricks.
- Mandatory: Practical experience with Monte Carlo Data Observability.
- Experience with Data Quality tools/frameworks.
- Hands-on experience with ETL/ELT pipelines.
- Experience with relational and NoSQL databases.
- Familiarity with Azure cloud data platforms.
- Familiarity with Collibra.
- Strong understanding of Data Warehouse, Data Lake, and Lakehouse architectures.
- Experience with batch and near-real-time data processing.
- Knowledge of dimensional and relational data modeling.
- Strong analytical and problem-solving skills.
Nice to Have
- Programming experience in Python, Scala, or Java.
- Experience with Airflow or Control-M.
- Knowledge of Data Governance, Metadata Management, and Data Lineage.
- Banking or other regulatory/compliance-driven data environment experience.
- CDMP, Azure, or AWS certifications.
Skills: azure,metadata management,data observability,data warehouse,data quality,data governance,aws,monte carlo,sql,etl,python,scala,nosql,java,databases
