SENIOR DATA ENGINEER - Databricks

Senior Data Engineer (68 years)Role Overview

Hands-on IC role focused on building reliable, performant data pipelines on Azure Databricks, implementing Delta Lake patterns, and contributing to production-grade data products for analytics.

Core Responsibilities

Build/maintain batch & basic streaming pipelines using PySpark/Spark SQL and Delta Lake following Medallion principles.

Ingest via Autoloader, ADF, REST, and file sources (e.g., SFTP/SharePoint).

Apply schema enforcement/evolution; add basic data quality checks and validations.

Use Unity Catalog for catalog/schema/table permissions; handle basic secrets via Key Vault/secret scopes.

Author clear code & docs; participate in PRs; contribute utilities; triage job issues and first-line performance fixes.

Operate simple Databricks Workflows (or ADF) and Git-based CI/CD basics.

Must-have Skills (Screening Focus)

Strong hands-on PySpark, Spark SQL, Delta Lake fundamentals; solid SQL.

Understanding of partitions, shuffles, joins, caching; Medallion basics.

Unity Catalog basics (catalog/schema/table permissions).

Autoloader basics; ADF familiarity or Workflows basics.

Git workflows; basic tests and logging; runbook hygiene.

Nice-to-have (Prefer, not Mandatory)

Exposure to DLT and Workflows orchestration patterns.

Photon awareness; cost-aware design exposure.

Healthcare/PII familiarity; Power BI integrations.

90-day Outcomes (Examples)

Ship 2 SilverGold pipelines with <1% failure rate; produce runbook.

Add basic data expectations (null, duplicate, type checks) with alerting on a critical table