MHR Azure DE Lead

Key Responsibilities:

• Data Architecture & Engineering
• Design and implement scalable data ingestion pipelines using Microsoft Fabric Data Factory (Dataflows) and OneLake.
• Build and manage Lakehouse and Warehouse architectures leveraging Delta Lake, Spark Notebooks, and SQL Endpoints.
• Define and enforce Medallion Architecture (Bronze, Silver, Gold) standards for structured data refinement and quality control.
• Data Modeling & Transformation
• Develop robust and reusable data transformation logic using PySpark, Scala, and Fabric SQL.
• Implement advanced data modeling strategies including:
• Slowly Changing Dimensions (SCD Type 2)
• Change Data Capture (CDC)
• Time-windowed aggregations and historical trend analysis
• Performance Optimization
• Monitor and optimize data pipeline performance with a focus on throughput, cost-efficiency, and reliability.
• Apply tuning techniques including partitioning, indexing, caching, and parallelism across Lakehouse and Warehouse compute.
• Data Quality & Governance
• Integrate Microsoft Purview for metadata management, lineage tracking, and data cataloging.
• Establish automated data quality rules, anomaly detection mechanisms, and proactive alerting systems.
• DevOps & Automation
• Implement Infrastructure-as-Code (IaC) using ARM templates or Terraform to provision Microsoft Fabric resources.
• Set up and maintain CI/CD pipelines using Git and Azure DevOps for deployment and version control across environments.
• Stakeholder Collaboration & Support
• Collaborate with data scientists, BI developers, and business stakeholders to understand requirements and deliver impactful data solutions.
• Provide production support, troubleshoot pipeline issues, and lead root-cause analysis to ensure system stability.

Requirements:

• 10-12 years of professional experience in data engineering , with a minimum of 1 year working hands-on with Microsoft Fabric .
• Proficiency in:
• Languages: SQL (T-SQL), Python, Scala
• Microsoft Fabric Components: Data Factory Dataflows, OneLake, Spark Notebooks, Lakehouse, Warehouse
• Data Formats: Delta Lake, Parquet, CSV, JSON
• Strong understanding of data modeling techniques: star schema , snowflake schema , normalized/denormalized structures .
• Experience in CI/CD practices and Infrastructure-as-Code (IaC) using Git , ARM templates , or terraform .
• Familiarity with data governance platforms such as Microsoft Purview .
• Excellent problem-solving and analytical skills with the ability to articulate complex technical concepts to diverse audiences.

Good To Have Skills:

• Microsoft Certified: Fabric Analytics Engineer or related certifications.
• Experience in agile development environments.
• Knowledge of cost management and optimization strategies in cloud environments.