Lead Data Engineer

Key Responsibilities:

• Design & Development:
• Lead the architecture and implementation of an enterprise-grade data platform using Microsoft Fabric and related Azure services.
• Design and implement Medallion architecture (Bronze/Silver/Gold layers) to manage raw, validated, and curated data.
• Develop real-time data ingestion, validation, and transformation pipelines supporting heterogeneous file formats.
• Build and orchestrate event-driven workflows using serverless Azure Functions, Event Hub, and Durable Functions.
• Integrate Fabric Lakehouses and Synapse Data Warehousing with transactional systems and UI components.
• Implement schema evolution, metadata management, and lineage tracking across the platform.
• Enable business self-service through configurable rules engines and validation UIs.
• Ensure role-based access control (RBAC), end-to-end encryption, and audit trails across all layers.
• Build CI/CD pipelines and monitoring for automated deployments and observability.
• Collaboration & Leadership:
• Collaborate with UI engineers, solution architects, product owners, and compliance stakeholders.
• Lead and mentor a team of data engineers; establish coding and documentation standards.

Required Skills and Qualifications:

• Experience:
• 8-12 years of hands-on experience in data engineering, including 3+ years in a leadership or architecture role.
• Core Skills:
• Strong expertise in Microsoft Azure, especially:
• Microsoft Fabric (Data Lakehouses, Pipelines, Real-Time Analytics, Synapse).
• Azure Data Factory, Azure Functions, Event Hub / Service Bus.
• Delta Lake on ADLS Gen2.
• Azure API Management and App Services.
• Advanced knowledge of Python, PySpark, SQL, and workflow orchestration.
• Experience building event-driven, serverless architectures in a production environment.
• Deep understanding of schema-agnostic ingestion, validation logic, and dynamic rule engines.
• Knowledge of data privacy, regulatory compliance, lineage, and auditability.
• Experience integrating with transactional UIs and REST APIs for CRUD operations.
• Proven track record delivering large-scale, cloud-native data platforms.

Good to Have:

• Domain knowledge in insurance, financial services, or regulatory systems.
• Familiarity with RE7/FSRI, master data integration, or third-party systems.
• Experience with retention, premium, or commission calculations.
• Prior exposure to Microsoft Purview, Azure Monitor, and Azure Key Vault.
• Understanding of multilingual support, global data residency compliance, and deployment automation.