Key Responsibilities
Data Architecture & Engineering
- Design, develop, and manage end-to-end data pipelines using Microsoft Fabric (Data Pipelines, Lakehouse, Warehouse, OneLake) and Azure Data Factory (ADF)
- Build scalable ETL/ELT solutions using PySpark, Python, SQL, and Fabric Notebooks
- Implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) for analytics workloads
Data Processing & Transformation
- Develop optimized SQL-based solutions including T-SQL, stored procedures, and performance-tuned queries
- Handle data transformation, enrichment, and processing across batch and near real-time pipelines
- Implement incremental loads, CDC (Change Data Capture), and watermarking techniques
Cloud Data Platform Engineering
- Work extensively with Azure Data Lake Storage Gen2, Azure Blob Storage, and OneLake
- Integrate data from multiple sources including RDBMS, APIs, streaming systems, and cloud storage
- Enable analytics and reporting through integration with Power BI and Fabric datasets
Streaming & Real-Time Data (Preferred)
- Work with streaming or near real-time data pipelines using Azure Event-based services
- Design event-driven data ingestion architectures where required
Data Governance & Quality
- Implement data quality checks, monitoring, logging, and error handling frameworks
- Ensure adherence to governance, security, and compliance standards
- Work with Microsoft Purview for data lineage and governance (preferred)
DevOps & CI/CD for Data Engineering
- Support CI/CD implementation for data pipelines using Azure DevOps and Git
- Collaborate with DevOps teams for deployment, version control, and release management
- Monitor pipeline performance and ensure reliability across environments
Migration & Modernization
- Support migration from Azure Data Factory and Databricks to Microsoft Fabric where applicable
- Contribute to enterprise-scale data platform modernization initiatives
Leadership & Collaboration
- Work closely with architects, analysts, and business stakeholders to translate requirements into scalable solutions
- Provide technical guidance and mentorship to junior engineers
- Drive best practices in data engineering, architecture, and performance optimization
