Senior Lead Engineer - ETL Testing

Job Summary

We are looking for an experienced QA professional with strong expertise in data analysis, ETL testing, data warehousing, database validation, and BI testing. The ideal candidate should have hands-on experience in Python-based data testing automation, SQL query validation, Azure Data Factory pipelines, Snowflake, and Power BI. The role involves end-to-end data validation, test automation, defect management, and collaboration with cross-functional teams to ensure data accuracy, completeness, and quality.

Key Responsibilities

• Perform end-to-end validation of ETL processes, data pipelines, data warehouses, and database systems.
• Validate data accuracy, completeness, consistency, and integrity across multiple systems.
• Test file ingestion processes and validate CSV, TXT, and Parquet files stored in Azure Data Lake Storage (ADLS).
• Execute and validate SQL queries across Snowflake, Azure SQL Managed Instance, and Amazon Redshift.
• Execute, monitor, and validate Azure Data Factory (ADF) pipelines and data movement across environments.
• Perform Power BI report testing and data validation.
• Develop and execute test strategies for data validation, UI testing, and REST API testing.
• Build and maintain Python-based test automation scripts for database validation, file handling, API testing, and Azure integration.
• Prepare test plans, test scenarios, test cases, test data, and test documentation.
• Manage the defect lifecycle, including identification, logging, tracking, reporting, and resolution.
• Track and report QA metrics and quality status to stakeholders.
• Participate in Agile estimation activities, including T-shirt sizing and Planning Poker.
• Provide technical guidance to team members and collaborate with developers, QA teams, and business stakeholders.

Mandatory Technical Skills

Data & ETL Testing

• ETL Testing and Data Pipeline Validation
• Data Warehousing and Database Testing
• Data Quality, Data Integrity, and Data Reconciliation
• File Ingestion and Validation
• CSV, TXT, and Parquet File Formats
• Power BI Testing and Validation

SQL & Database Technologies

• Advanced SQL Query Writing and Optimization
• Snowflake
• Azure SQL Managed Instance
• Amazon Redshift
• Database Validation Across Multiple Platforms

Python & Test Automation

• Python-Based Data Testing Automation
• Python Libraries: pandas, pyodbc, requests, and SQLAlchemy
• Database Cursors and Database Connectivity
• File Handling and Data Comparison
• REST API Testing
• Azure SDKs

Azure & DevOps

• Microsoft Azure Cloud Services
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• Azure DevOps Pipelines
• Azure DevOps Test Plans
• Pipeline Execution and Validation

QA & Testing Practices

• Test Planning and Test Case Design
• UI Testing and REST API Testing
• Defect Management and Reporting
• Regression Testing
• QA Metrics and Test Documentation
• Agile/Scrum Methodologies

Good-to-Have Skills

• Pytest and Robot Framework
• Data Testing Automation Framework Development
• Automated Regression Test Suites
• Data Testing Across On-Premises and Azure Cloud Environments
• Advanced Snowflake Testing
• Azure DevOps Test Plans and Pipeline Automation
• Experience with reusable and scalable test automation frameworks

Candidate Requirements

• 10+ years of experience in QA, data analysis, ETL, data warehouse, BI, and data quality testing.
• Strong hands-on experience in Python, SQL, and data validation across multiple databases.
• Experience testing Azure Data Factory pipelines and validating files in ADLS.
• Strong understanding of Power BI testing, UI testing, and REST API testing.
• Experience designing test strategies, preparing test data, and managing defects.
• Strong analytical, troubleshooting, communication, and stakeholder collaboration skills.
• Ability to guide team members and work effectively in Agile delivery environments.