SENIOR DATA ENGINEER - Databricks

Senior Data Engineer (6–8 years)Role Overview

Hands-on IC role focused on building reliable, performant data pipelines on Azure Databricks, implementing Delta Lake patterns, and contributing to production-grade data products for analytics.

Core Responsibilities

• Build/maintain batch & basic streaming pipelines using PySpark/Spark SQL and Delta Lake following Medallion principles.

• Ingest via Autoloader, ADF, REST, and file sources (e.g., SFTP/SharePoint).

• Apply schema enforcement/evolution; add basic data quality checks and validations.

• Use Unity Catalog for catalog/schema/table permissions; handle basic secrets via Key Vault/secret scopes.

• Author clear code & docs; participate in PRs; contribute utilities; triage job issues and first-line performance fixes.

• Operate simple Databricks Workflows (or ADF) and Git-based CI/CD basics.

Must-have Skills (Screening Focus)

• Strong hands-on PySpark, Spark SQL, Delta Lake fundamentals; solid SQL.

• Understanding of partitions, shuffles, joins, caching; Medallion basics.

• Unity Catalog basics (catalog/schema/table permissions).

• Autoloader basics; ADF familiarity or Workflows basics.

• Git workflows; basic tests and logging; runbook hygiene.

Nice-to-have (Prefer, not Mandatory)

• Exposure to DLT and Workflows orchestration patterns.

• Photon awareness; cost-aware design exposure.

• Healthcare/PII familiarity; Power BI integrations.

90-day Outcomes (Examples)

• Ship 2 SilverGold pipelines with <1% failure rate; produce runbook.

• Add basic data expectations (null, duplicate, type checks) with alerting on a critical table