Lead Software Engineer

Job Summary: We are looking for a highly skilled Senior Azure Databricks Engineer to lead the design, development, and optimization of our enterprise Data Lakehouse. This role bridges the gap between traditional data engineering and modern cloud scale, focusing on building high-performance, reliable, and secure data pipelines. You will be instrumental in implementing the Medallion Architecture and ensuring that our data ecosystem supports both real-time streaming and complex batch analytics for downstream consumers.

Key Responsibilities: Lakehouse Development: Design and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL in both batch and streaming modes. Medallion Architecture: Implement and govern the Bronze, Silver, and Gold layers using Delta Lake, ensuring ACID compliance, schema enforcement, and data versioning (Time Travel). Performance Engineering: Optimize Spark jobs by managing partitions, caching strategies, and broadcast joins. Leverage Adaptive Query Execution (AQE) and Z-Ordering to minimize compute latency and cost. Stream Processing: Build and manage seamless data flows using Apache Kafka or Confluent to support real-time data ingestion and event-driven architectures. Governance & Security: Configure and manage Unity Catalog for fine-grained access control, automated data lineage, and comprehensive audit logging. Orchestration: Automate complex workflows using Databricks Workflows, Delta Live Tables (DLT), or Azure Data Factory (ADF). Collaboration & Mentorship: Partner with Data Architects and Scientists to translate business logic into technical specifications. Provide technical mentorship to junior engineers and drive engineering best practices.

Required Technical Skills: Databricks Ecosystem: Mastery of the Databricks Lakehouse Platform, including Delta Lake, Unity Catalog, and Photon engine. Programming: Advanced proficiency in Python (PySpark) and SQL for complex data transformations; exposure to Scala is a plus. Streaming & Integration: Proven experience with Kafka and integrating Databricks with Azure services like ADLS Gen2, Cosmos DB, and Azure Key Vault. Database & Storage: Deep knowledge of NoSQL databases (Cosmos DB, MongoDB) and file formats like Parquet and Avro.DevOps & CI/CD: Familiarity with Git-based workflows, Terraform for Infrastructure as Code (IaC), and automated deployment pipelines.