Search by job, company or skills

Happiest Minds Technologies

LEAD DATA ENGINEER - Databricks

Happiest Minds Technologies
Bengaluru, India
8-10 Years
Save
new job description bg glownew job description bg glow
  • Posted 11 hours ago
  • Be among the first 10 applicants
Early Applicant

Job Description

  • C4 – Lead Data Engineer (8–10 years)
  • Role Overview
  • Engineering lead who owns design & delivery end-to-end; raises bar on performance, DataOps, and governance; mentors C3s.
  • Core Responsibilities
    • Design high-throughput batch & streaming (Structured Streaming/Autoloader) to Medallion targets.
    • Implement advanced Delta: MERGE/SCD2, schema evolution/enforcement, OPTIMIZE/Z-ORDER/VACUUM, compaction.
    • Engineer observability: metrics, alerts, SLIs/SLOs, error budgets; on-call runbooks.
    • Implement Unity Catalog patterns (RBAC/ABAC), row/column-level security; Key Vault integration; PII/PHI handling.
    • Own cost & performance: Photon, autoscaling, cluster policies/pools; job SLAs.
    • Lead CI/CD (Repos/CLI/ADO) with automated unit/data tests and promotion gates; design reviews and reusable frameworks.
  • Must-have Skills (Screening Focus)
    • Deep PySpark/Spark SQL/Delta; DLT + Workflows orchestration.
    • MERGE/SCD2; schema evolution/enforcement; time travel.
    • Performance/cost levers: OPTIMIZE/Z-ORDER, file sizing, Photon, autoscaling; cluster policies/pools.
    • Unity Catalog RBAC/ABAC; secrets; PII/PHI controls.
    • CI/CD with tests (unit/data expectations) and environment promotion; on-call readiness.
  • Nice-to-have (Prefer, not Mandatory)
    • Kafka/Event Hubs streaming patterns; Synapse integration.
    • Delta Sharing, Lakehouse Federation; dbt for transformations.
  • 90-day Outcomes (Examples)
    • Reduce a priority job's runtime ≥25% and cost/GB ≥20% within 90 days.
    • Stand up CI/CD + tests + SLOs for one DLT pipeline with alerting; publish a reusable SCD2/DQ module

More Info

Job Type:
Permanent Job
Industry:
Other
Function:
Data Engineering
Employment Type:
Full time

Job ID: 147461367

Jobs by Skill - IT
Jobs by Skill - Non IT
International Jobs
Last Updated: 15-05-2026 06:17:57 PM
Homejobs in Bengaluru / BangaloreLEAD DATA ENGINEER - Databricks

Similar Jobs

Lead Data Engineer (Databricks)

Rearc
Bengaluru, India

Skills:

Pyspark, Java, Aws Redshift, AWS Glue, Data Warehousing, Data Architecture, Numpy, Pandas, Azure Synapse Analytics, Spark, Databricks, Data Modelling, Python, Airflow, Google BigQuery, Data integration tools, dbt, ETL processes, Dask, Delta Lake, Data pipeline orchestration