Azure Databricks Support Engineer (L2/L3)

Job Title: Azure Databricks Support Engineer (L2/L3)

Job Summary

We are looking for an Azure Databricks Support Engineer with strong experience in Databricks (ADB), PySpark, and Python scripting, who can manage data platform infrastructure and provide 24×7 production support. The role involves a mix of infrastructure operations (60%) and development/support automation (40%) to ensure platform stability, performance, and reliability.

Key Responsibilities

Databricks & Data Platform Support

• Provide L2/L3 production support for Azure Databricks workloads and data pipelines
• Monitor, troubleshoot, and resolve issues related to Databricks jobs, clusters, notebooks, and workflows
• Handle incident management, root cause analysis (RCA), and problem management
• Ensure adherence to SLAs and operational KPIs

Infrastructure & Cloud Operations (60%)

• Manage and support Azure Databricks infrastructure, including cluster configuration, scaling, and optimization
• Monitor system health, performance, and availability using monitoring/logging tools
• Support deployment, upgrades, and patching of Databricks environments
• Work closely with cloud, security, and networking teams

Development & Automation (40%)

• Develop and maintain Python & PySpark scripts for data processing and automation
• Support and enhance existing ETL/ELT pipelines
• Automate operational tasks to improve platform reliability and efficiency
• Perform performance tuning and optimization of Spark jobs

Support & Collaboration

• Participate in 24×7 rotational shift support and on-call rotations
• Collaborate with data engineers, platform teams, and business stakeholders
• Create and maintain support documentation, SOPs, and runbooks

Required Skills & Experience

Technical Skills

• Strong experience with Azure Databricks (ADB)
• Hands-on experience in Python and PySpark
• Solid understanding of Spark architecture and performance tuning
• Experience with Azure cloud services
• Familiarity with production support processes (ITIL-based)

Experience

• 3–7 years of experience in data engineering, big data support, or data platform operations
• Experience in 24×7 production support environments is mandatory

Good to Have

• Experience with CI/CD pipelines
• Knowledge of monitoring tools (Azure Monitor, Log Analytics, etc.)
• Exposure to DevOps or SRE practices
• Experience in regulated or large enterprise environments

Work Model

• 24×7 rotational shifts
• Hybrid / Onsite / Remote (as applicable)