Data Engineer

Role Overview

We are looking for a hands-on, senior Databricks Architect to design, build, and govern our Lakehouse data platform from the ground up. You will own the end-to-end architecture of our data infrastructure from raw ingestion through the Medallion layers to serving and establish the engineering standards that will guide the entire data organization.

This is a highly strategic and technical role focused on driving adoption of Databricks, Unity Catalog, and modern Lakehouse patterns across all data products and pipelines.

Key Responsibilities

Lakehouse Architecture & Design

• Design and implement a production-grade Medallion Architecture (Bronze / Silver / Gold) across all data pipelines.
• Establish best practices for Delta Lake table design, partitioning strategies, Z-ordering, and optimization across large-scale datasets.
• Define data modeling standards and schema evolution policies across the Lakehouse.
• Architect end-to-end data flows from ingestion (streaming and batch) through transformation and serving layers.

Unity Catalog & Data Governance

• Lead the setup, configuration, and rollout of Unity Catalog as the centralized governance layer for all data assets.
• Design metastore hierarchy, catalog/schema/table organization, and tagging standards.
• Implement fine-grained access control (row-level, column-level), data masking policies, and audit logging.
• Establish data lineage tracking and ensure end-to-end visibility across all pipelines.
• Define and enforce data classification and sensitivity frameworks for PII and regulated data assets.

Pipeline Development & Orchestration

• Build and maintain production-grade data pipelines using PySpark, Delta Live Tables (DLT), and Databricks Workflows / Jobs.
• Design modular, reusable pipeline patterns including incremental ingestion, CDC (Change Data Capture), and full-refresh strategies.
• Implement robust pipeline observability: logging, alerting, lineage tracking, and SLA monitoring.
• Leverage Databricks Repos for CI/CD integration, managing code promotion across dev / staging / production environments.

Performance & Compute Optimization

• Optimize Spark execution plans, identify and resolve performance bottlenecks across large-scale distributed workloads.
• Right-size cluster configurations: Serverless warehouses, auto-scaling job clusters, and photon-enabled SQL warehouses.
• Leverage Serverless Warehouses and SQL Warehouses for BI and ad hoc analytics workloads, minimizing cost and cold-start latency.
• Manage cost governance for compute, storage, and DBU consumption across workspaces.

Developer Experience & Standards

• Set up and maintain Databricks Repos with standardized project structures and Git integration.
• Define Python coding standards, notebook best practices, and modular library patterns for the data engineering team.
• Build reusable Python utility libraries for common patterns: schema validation, data quality checks, Delta operations, and logging.
• Establish unit testing and integration testing frameworks for Spark pipelines.

Security, Compliance & Networking

• Configure workspace-level and account-level security: Private Link, IP access lists, secrets management via Databricks Secrets or AWS Secrets Manager.
• Design and enforce network isolation for sensitive data workloads.
• Ensure compliance with data residency and access control requirements for customer data.

Collaboration & Enablement

• Partner with data engineers, data scientists, and analytics engineers to ensure the platform meets diverse workload needs.
• Mentor the engineering team on Databricks, Spark optimization, and Lakehouse best practices.
• Produce architectural documentation, runbooks, and internal knowledge bases.
• Evaluate and recommend new Databricks features and third-party integrations relevant to the organization's data roadmap.

Required Qualifications

Core Databricks & Lakehouse

• 5+ years of hands-on experience with Databricks, with at least 2 years in an architect or senior lead role.
• Deep expertise in Unity Catalog: metastore setup, three-level namespace, ACL design, and data governance workflows.
• Strong mastery of the Medallion Architecture and Delta Lake: ACID transactions, time travel, compaction, and OPTIMIZE/VACUUM strategies.
• Proven experience designing and deploying production pipelines with Databricks Jobs and Workflows, including multi-task job DAGs, retry logic, and notifications.
• Hands-on experience with Databricks Repos and CI/CD integration for notebook and Python library deployments.
• Experience configuring and operating Serverless SQL Warehouses and Serverless compute for Jobs.

Apache Spark

• Expert-level PySpark development: DataFrames, Spark SQL, window functions, broadcast joins, and UDFs.
• Strong understanding of Spark internals: DAG execution, shuffle optimization, memory management, and speculative execution.
• Experience with structured streaming and micro-batch processing patterns.
• Proven ability to diagnose and resolve Spark performance issues using Spark UI and event logs.

Python & Software Engineering

• Advanced Python skills with a strong software engineering background: packaging, testing (pytest), virtual environments, and dependency management.
• Experience building modular Python libraries for data engineering use cases.
• Familiarity with common data engineering libraries: pandas, pydantic, great_expectations or similar DQ frameworks.

Cloud & Infrastructure

• Experience deploying Databricks on AWS, including workspace provisioning, IAM integration, and VPC configuration.
• Familiarity with cloud-native storage (S3/ADLS), external locations in Unity Catalog, and storage credentials management.
• Exposure to infrastructure-as-code tooling (Terraform, Databricks Asset Bundles, or similar).

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Certified Associate Developer for Apache Spark certifications.
• Experience with Delta Live Tables (DLT) for declarative pipeline authoring.
• Familiarity with dbt (data build tool) integrated with Databricks SQL.
• Experience with Databricks Feature Store or MLflow for ML platform use cases.
• Exposure to Databricks Marketplace and Partner Connect integrations.
• Experience with Elasticsearch, Apache Kafka, or other streaming/search technologies complementary to the Lakehouse.