Lead Data Engineer HIP Integration

Lead Data Engineer – HIP Integration

Lead Data Engineer – HIP Integration

Experience: 8+ years

Location: As per requirement

Role: Lead Data Engineer

Key Responsibilities:

• Own end-to-end onboarding of CMS/AS data into the Heathrow Insights Platform.
• Design and implement Databricks Lakehouse solutions using Delta Lake, Unity Catalog, Lakeflow Jobs and Bronze/Silver/Gold architecture.
• Select the right integration pattern based on data and business requirements.
• Build batch, micro-batch and near-real-time data pipelines.
• Lead a team of 2 Data Engineers working on CMS and AS integrations.
• Ensure engineering standards around reusability, discoverability, observability, documentation and technical debt reduction.
• Work closely with data product owners across Baggage, Airside Operations, Carbon, Forecasting and Commercial Finance.
• Prepare data for consumption in Gold layer and Power BI reports.

Must-Have Skills:

• Databricks
• Unity Catalog
• Delta Lake
• Lakeflow Jobs / notebook orchestration
• Medallion Architecture – Bronze, Silver, Gold
• Azure / Event Hubs / Kafka
• Batch & near-real-time data engineering
• API ingestion – REST / GraphQL
• Strong understanding of integration patterns

Integration Patterns:

1. Direct/Streaming: Event Hubs Kafka endpoint, Kafka, Zerobus
2. Lakeflow Connect: Salesforce, SQL Server, Azure SQL, SFTP, Snowflake Federation
3. Open Sharing
4. Custom REST/GraphQL orchestration

Leadership Skills:

• Team leadership
• Technical decision-making
• Stakeholder management
• Engineering standards & governance
• Strong communication and documentation