Azure Data Lake / ETL QA tester (5:00 PM - 2:00 AM)

Position Summary

We are looking for an experienced Azure Data Lake / ETL QA Tester to support end-to-end testing of cloud-based data platforms and data pipelines built on Microsoft Azure.

The candidate will be responsible for validating data ingestion, transformation, storage, reconciliation, and downstream consumption across Azure Data Factory, Azure Data Lake Storage, Azure Databricks, Azure SQL/Synapse, and reporting platforms.

The ideal candidate should have strong experience in ETL/Data Warehouse testing, SQL, Azure Data Factory, Azure Data Lake, Databricks, Python/PySpark, and source-to-target data validation.

Key Responsibilities

Review business requirements, data mappings, source-to-target specifications, transformation rules, and technical design documents.

Develop detailed test scenarios and test cases for ETL, Data Lake, and Data Warehouse solutions.

Perform end-to-end validation of data pipelines from source systems through ingestion, transformation, curated layers, and downstream reporting.

Validate data ingestion into Azure Data Lake Storage Gen2 (ADLS).

Test Azure Data Factory pipelines, including:

Pipeline execution

Triggers and dependencies

Incremental and full loads

Error handling and retries

Restart and recovery scenarios

Validate data transformations performed in Azure Databricks using SQL, Spark SQL, Python, and PySpark.

Perform source-to-target reconciliation and validate:

Record counts

Data completeness

Data accuracy

Duplicate records

Null values

Referential integrity

Transformation rules

Validate data across Bronze, Silver, and Gold layers in a Medallion Architecture.

Test full loads, incremental loads, historical loads, and CDC-based processing.

Validate SCD Type 1 and Type 2 transformations where applicable.

Validate Delta Lake tables, schemas, partitions, audit fields, timestamps, and historical data.

Perform negative, boundary, exception, integration, regression, and data-volume testing.

Develop reusable SQL, Python, and PySpark validation scripts to improve test efficiency and automation.

Validate downstream reporting and analytics data, including Power BI where applicable.

Log, track, and manage defects using Azure DevOps, Jira, qTest, or equivalent tools.

Participate in defect triage, root-cause analysis, retesting, and regression testing.

Maintain test evidence, traceability, test execution results, and quality metrics.

Participate in Agile ceremonies, sprint planning, release-readiness reviews, and production deployment support.

Work closely with Data Engineers, Architects, Business Analysts, Product Owners, Developers, and QA teams.

Required Qualifications

Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.

Minimum 5 years of experience in ETL, Data Warehouse, Data Lake, or Data Platform testing.

Strong hands-on experience with SQL and complex data validation queries.

Hands-on experience with Azure Data Factory (ADF).

Experience testing data stored in Azure Data Lake Storage Gen2.

Experience with Azure Databricks.

Working knowledge of Python and/or PySpark.

Strong experience with source-to-target data reconciliation.

Understanding of ETL/ELT concepts, Data Warehouse, Data Lake, and Lakehouse architectures.

Experience validating Bronze, Silver, and Gold data layers.

Experience with data-quality validation including completeness, accuracy, consistency, duplicates, nulls, and referential integrity.

Experience with Agile/Scrum delivery methodologies.

Experience with defect and test-management tools such as Azure DevOps, Jira, or qTest.

Preferred Qualifications

Experience with Delta Lake and Unity Catalog.

Experience with Azure Synapse Analytics.

Experience with Snowflake or similar cloud data warehouses.

Experience validating Power BI reports and downstream analytics.

Experience with automated data-testing frameworks.

Experience with CI/CD pipelines, Git, GitHub, or Azure DevOps pipelines.

Exposure to Tricentis Tosca or other enterprise test-automation tools.

Experience with metadata-driven pipelines, control tables, audit frameworks, and large-volume data reconciliation.

Key Technical Skills

Mandatory:

Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, SQL/T-SQL, ETL Testing, Data Warehouse Testing, Source-to-Target Validation, Data Reconciliation, Python/PySpark, Data Quality Testing.

Preferred:

Delta Lake, Unity Catalog, Azure Synapse, Snowflake, Power BI, CI/CD, Azure DevOps, qTest, Tosca.

Key Competencies

Strong analytical and problem-solving skills

Ability to analyze complex data transformations and business rules

Strong attention to data quality and accuracy

Ability to work independently and within cross-functional teams

Strong communication and defect-management skills

Ability to manage multiple testing priorities in an Agile environment

Success Criteria

The candidate will be expected to ensure that data delivered through the Azure platform is complete, accurate, traceable, reconciled, and fit for downstream business and reporting consumption, while continuously improving test coverage and automation.