ETL+ Azure services ((like azure databricks/ data factory)
Description :
- Experienced in end-to-end of ETL (Data Pipeline) QA testing using various technologies in Azure cloud involving ingestion from varied data sources.
- Should be familiar with data warehousing target and intermediate structures ideally in environments using data lakes. Well versed in Databricks, SQL Database, Azure Data Factory, Azure Data Lake
- Must have hands-on experience in PySpark, SparkSQL, T-SQL
- Should be able to go through requirements, design specs and create appropriate test scenarios across various layers/stages of data pipelines.
- Should be able to write reusable PySpark and T-SQL script to validate data across layers.
- Experience of working in Databricks Delta Lake will be highly desirable.
- Ideally should have worked in a team using Agile methodology.
- Appreciation of data, its quality, and its usage for business benefit
- Very good communication and written skills
