Data Engineer

Job Title: Data Engineers

Experience Required: 3-5 Years & 6-10 Years

Locations: Navi Mumbai

Duration; Fulltime

Positions: Multiple

Job Summary: We are looking for a highly skilled Azure Data Engineer with a strong background in real-time and batch data ingestion and big data processing, particularly using Kafka and Databricks. The ideal candidate will have a deep understanding of streaming architectures, Medallion data models, and performance optimization techniques in cloud environments. This role requires hands-on technical expertise, including live coding during the interview process.

Key Responsibilities

• Design and implement streaming data pipelines integrating Kafka with Databricks using Structured Streaming.

• Architect and maintain Medallion Architecture with well-defined Bronze, Silver, and Gold layers.

• Implement efficient ingestion using Databricks Autoloader for high-throughput data loads. • Work with large volumes of structured and unstructured data, ensuring high availability and performance.

• Apply performance tuning techniques such as partitioning, caching, and cluster resource optimization.

• Collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions.

• Establish best practices for code versioning, deployment automation, and data governance.

Required Technical Skills:

• Strong expertise in Azure Databricks and Spark Structured Streaming

• 3-8 Years experience in Data Engineering

• Processing modes (append, update, complete)

• Output modes (append, complete, update)

• Checkpointing and state management

• Experience with Kafka integration for real-time data pipelines

• Deep understanding of Medallion Architecture

• Proficiency with Databricks Autoloader and schema evolution

• Deep understanding of Unity Catalog and Foreign catalog

• Strong knowledge of Spark SQL, Delta Lake, and DataFrames

• Expertise in performance tuning (query optimization, cluster configuration, caching strategies)

• Must have Data management strategies

• Excellent with Governance and Access management

• Strong with Data modelling, Data warehousing concepts, Databricks as a platform

• Solid understanding of Window functions