Data Engineer (Databricks, Python)

Hiring for a Client - KnackForge

Key Responsibilities

• Architect and implement enterprise-grade Lakehouse solutions using Databricks
• Design and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL)
• Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworks
• Implement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and Workflows
• Design large-scale data warehousing solutions with 3NF and dimensional modeling
• Establish data governance, security, and data quality frameworks, including Unity Catalog
• Lead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI)
• Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow)
• Drive CI/CD, DevOps practices, and performance optimization of Spark workloads
• Provide technical leadership, mentor teams, and ensure successful delivery
• Collaborate with stakeholders to translate business requirements into scalable solutions

Ideal Candidate — Databricks Lead

• 10+ years of total experience, with 5+ years specifically in Databricks
• Architecture-level expertise — must understand end-to-end design, not just engineering execution
• Hands-on with Delta Lake, Delta Live Tables, Spark (PySpark/SQL), and Azure Data Factory
• Prior experience working on Azure or Azure DevOps environments
• Has handled US/UK or international client-facing projects
• Comfortable managing and leading a team
• Basic understanding of AI/ML tools and services is a plus

Perks, Benefits & Culture

• The company provides free AWS and Azure certification training
• Group medical insurance of ₹5 lakhs is included in the benefits package, along with meal allowances