LEAD DATA ENGINEER - Azure Databricks

Role Overview

Engineering lead who owns design & delivery end-to-end, raises bar on performance, DataOps, and governance, mentors team members. 

Core Responsibilities

·       Design high-throughput batch & streaming (Structured Streaming/Autoloader) to Medallion targets.

·       Implement advanced Delta: MERGE/SCD2, schema evolution/enforcement, OPTIMIZE/Z-ORDER/VACUUM, compaction.

·       Engineer observability: metrics, alerts, SLIs/SLOs, error budgets; on-call runbooks.

·       Implement Unity Catalog patterns (RBAC/ABAC), row/column-level security; Key Vault integration; PII/PHI handling.

·       Own cost & performance: Photon, autoscaling, cluster policies/pools; job SLAs.

·       Lead CI/CD (Repos/CLI/ADO) with automated unit/data tests and promotion gates; design reviews and reusable frameworks.

Must-have Skills (Screening Focus)

·       Deep PySpark/Spark SQL/Delta; DLT + Workflows orchestration.

·       MERGE/SCD2; schema evolution/enforcement; time travel.

·       Performance/cost levers: OPTIMIZE/Z-ORDER, file sizing, Photon, autoscaling; cluster policies/pools.

·       Unity Catalog RBAC/ABAC; secrets; PII/PHI controls.

·       CI/CD with tests (unit/data expectations) and environment promotion; on-call readiness.

Nice-to-have (Prefer, not Mandatory)

·       Kafka/Event Hubs streaming patterns; Synapse integration.

·       Delta Sharing, Lakehouse Federation; dbt for transformations.