Design, develop, and maintain automated ETL testing frameworks and validation workflows across data pipelines.
Perform comprehensive testing of ETL processes in Databricks, Azure Data Lake, and Azure Synapse environments, ensuring data accuracy, completeness, and consistency.
Build and optimize automated test suites for data validation, transformation logic, and performance testing using tools such as PySpark, Selenium, pytest, or other test automation frameworks.
Develop and maintain data quality and monitoring scripts to ensure the integrity of large-scale datasets and timely detection of data anomalies.
Collaborate with data engineers to validate pipeline performance, error handling, and data lineage.
Work with DevOps team to Implement CI/CD-based automated testing and reporting for ETL workflows in integration with DevOps pipelines.
Ensure compliance with security, data governance, and RBAC policies within the data testing framework.
Troubleshoot issues in data ingestion, transformation, and validation processes, working closely with development teams for resolution.
Mentor junior QA engineers and contribute to test strategy, tooling, and process improvements across the data platform.
