Job Title
Technical Lead – Microsoft Fabric Data Engineering
Location
Noida, India
Overview of the Role
As a Technical Lead – Microsoft Fabric Data Engineering, you will spearhead the design and implementation of advanced data solutions that harness the capabilities of Microsoft Fabric. Your expertise will drive robust, scalable, and high-quality data pipelines, enabling actionable business intelligence and supporting HCLTech's strategic transformation initiatives. You will play a critical role in ensuring data integrity, optimizing performance, and fostering a data-driven culture across the organization.
Detailed Responsibilities
- Architect, develop, and implement scalable ETL (Extract, Transform, Load) processes using Microsoft Fabric, integrating data from diverse sources into cloud-based data warehouses and lakehouses.
- Collaborate closely with data analysts, engineers, and business stakeholders to define requirements and translate them into effective and sustainable data solutions.
- Build, maintain, and optimize Power BI dashboards and reports, transforming complex datasets into actionable insights for informed business decisions.
- Ensure data quality, integrity, and accuracy throughout the ETL lifecycle, adhering to industry best practices for data management and governance.
- Monitor, troubleshoot, and optimize ETL processes, proactively identifying and resolving issues or performance bottlenecks to maintain high system reliability.
- Document ETL workflows, data mappings, and process modifications to facilitate transparency and clear communication across technical and non-technical teams.
- Implement advanced data transformation and ingestion patterns, including Change Data Capture (CDC), schema evolution, error handling, and robust data operations.
- Uphold best practices in data security, privacy, and compliance throughout all data engineering activities.
Skill Requirements
Must-Have Skills & Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
- 7+ years of hands-on experience in ETL development, with recent projects utilizing Microsoft Fabric.
- Expertise in Microsoft Fabric Data Pipelines (activities, triggers, parameters) and Notebooks (PySpark/Spark SQL).
- Deep working knowledge of Delta Lake, including schema evolution, MERGE/UPSERT, OPTIMIZE/VACUUM, partitioning, and checkpointing.
- Strong proficiency in CDC ingestion patterns (watermarks, change tables, log-based ingestion).
- Advanced SQL skills (window functions, CTEs) and experience with PySpark for data transformations and performance tuning.
- Demonstrated experience in implementing data quality frameworks: rule patterns (null, range, referential, uniqueness), scorecards, and exception handling.
- Advanced error handling and observability: idempotency, retries, dead-letter handling, and alerting hooks.
- Solid understanding of business intelligence concepts with practical experience in Power BI for data visualization and reporting.
- Knowledge of data warehousing concepts, architectures, and modeling techniques.
- Familiarity with major cloud platforms (Azure, AWS, Google Cloud) and their respective data storage and processing services.
- Excellent analytical, troubleshooting, and problem-solving skills.
- Strong communication skills, with the ability to convey complex technical concepts to non-technical audiences.
Good-to-Have Skills
- Experience with additional ETL tools (e.g., SSIS) and data warehousing solutions.
- Understanding of data governance, data privacy, and compliance standards.
- Familiarity with programming languages such as Python or R for advanced data manipulation.
- Experience working with stored procedures within Fabric transformations.
- Knowledge of metadata-driven pipelines and parameterization (YAML/JSON controlled).
- Awareness of Power BI model readiness (star schema, aggregations).
Other Requirements (Optional)
- Professional certifications in cloud platforms (e.g., Azure Data Engineer Associate, AWS Certified Data Analytics) are advantageous.
- Experience in large-scale enterprise environments is a plus.
- Demonstrated ability to lead and mentor junior engineers or cross-functional teams.
