Job Title : Senior QA Engineer Microsoft Fabric (Data Engineering & Notebook Validation)
Role Summary
We are seeking an experienced Senior QA Engineer (7+ years) with strong expertise in Microsoft Fabric, particularly within the Data Engineering and Data Analytics environment.
The Ideal Candidate Will Have Hands-on Experience In
- Developing and testing notebooks in Fabric (PySpark / Spark SQL)
- Supporting and validating data migrations to Microsoft Fabric
- Designing automated data validation frameworks
- Ensuring data integrity across Lakehouse, pipelines, and Power BI environments
This role requires a technically strong QA professional who understands both backend data validation and automation within modern data platforms.
Key Responsibilities
Microsoft Fabric & Data Engineering QA :
- Perform end-to-end validation of data pipelines within Microsoft Fabric.
- Validate data across OneLake, Lakehouse, and Fabric SQL endpoints.
- Test Bronze, Silver, and Gold data layers for completeness, accuracy, and transformation integrity.
- Collaborate with Data Engineers to validate transformation logic and incremental load processing.
Notebook Development & Validation
- Develop and execute validation logic within Fabric notebooks using PySpark and Spark SQL.
- Perform schema validation, row count reconciliation, null checks, duplicate checks, and aggregation validation.
- Build reusable notebook-based validation frameworks.
- Log validation results into Delta tables for auditability and traceability.
Data Migration To Microsoft Fabric
- Validate data migrations from legacy platforms (AWS, Azure, On-prem, etc.) to Microsoft Fabric.
- Perform before-and-after data comparison between legacy and Fabric environments.
- Validate transformation logic, metadata migration, and incremental load accuracy.
- Support report-level validation post-migration (Power BI comparisons).
QA Automation In Data Platforms
- Design and implement automated validation frameworks for data pipelines.
- Integrate notebook-based validations into Fabric pipelines.
- Support CI/CD integration for automated QA gating.
- Develop automated checks for :
- Data completeness
- Schema consistency
- Business rule validation
- Incremental load verification
Collaboration & Reporting
- Work closely with Data Engineers, BI Developers, and Business Analysts.
- Perform root cause analysis for data discrepancies.
- Document test cases, validation logic, and QA metrics.
- Provide QA sign-off for production releases.
Required Qualifications
- 7+ years of experience in Data QA, ETL Testing, or BI/Data Engineering validation.
- Strong hands-on experience in Microsoft Fabric (Data Engineering environment).
- Proven experience writing and executing Fabric notebooks (PySpark / Spark SQL).
- Experience supporting data migration projects to Microsoft Fabric.
- Strong SQL skills for backend data validation.
- Experience validating Delta/Parquet data structures.
- Understanding of data quality principles :
- Accuracy
- Completeness
- Consistency
- Timeliness
- Experience designing or implementing QA automation in data platforms.
- Exposure to CI/CD pipelines and DevOps integration for data testing.
Preferred Qualifications
- Experience validating Power BI reports post Fabric migration.
- Knowledge of OneLake architecture and Delta Lake format.
- Experience in Azure Data ecosystem.
- Exposure to Python scripting for data validation.
- Experience in regulated or enterprise data environments.
Key Technical Skills
- Microsoft Fabric (Lakehouse, OneLake, Fabric SQL)
- PySpark
- Spark SQL
- Advanced SQL
- Delta Lake
- Data migration validation
- QA automation frameworks
- CI/CD integration
- Power BI validation (preferred)
What Success Looks Like
- Successful validation of Fabric migrations with zero critical data defects.
- Automated notebook-based data validation framework in place.
- Reduced manual data testing effort through automation.
- High confidence in data accuracy across reporting and analytics layers.
(ref:hirist.tech)
