Azure Databricks Support Engineer (L2/L3)

Job Title: Azure Databricks Support Engineer (L2/L3)

Job Summary

We are looking for an Azure Databricks Support Engineer with strong experience in Databricks (ADB), PySpark, and Python scripting, who can manage data platform infrastructure and provide 247 production support. The role involves a mix of infrastructure operations (60%) and development/support automation (40%) to ensure platform stability, performance, and reliability.

Key Responsibilities

Databricks & Data Platform Support

• Provide L2/L3 production support for Azure Databricks workloads and data pipelines
• Monitor, troubleshoot, and resolve issues related to Databricks jobs, clusters, notebooks, and workflows
• Handle incident management, root cause analysis (RCA), and problem management
• Ensure adherence to SLAs and operational KPIs

Infrastructure & Cloud Operations (60%)

• Manage and support Azure Databricks infrastructure, including cluster configuration, scaling, and optimization
• Monitor system health, performance, and availability using monitoring/logging tools
• Support deployment, upgrades, and patching of Databricks environments
• Work closely with cloud, security, and networking teams

Development & Automation (40%)

• Develop and maintain Python & PySpark scripts for data processing and automation
• Support and enhance existing ETL/ELT pipelines
• Automate operational tasks to improve platform reliability and efficiency
• Perform performance tuning and optimization of Spark jobs

Support & Collaboration

• Participate in 247 rotational shift support and on-call rotations
• Collaborate with data engineers, platform teams, and business stakeholders
• Create and maintain support documentation, SOPs, and runbooks

Required Skills & Experience

Technical Skills

• Strong experience with Azure Databricks (ADB)
• Hands-on experience in Python and PySpark
• Solid understanding of Spark architecture and performance tuning
• Experience with Azure cloud services
• Familiarity with production support processes (ITIL-based)

Experience

• 37 years of experience in data engineering, big data support, or data platform operations
• Experience in 247 production support environments is mandatory

Good to Have

• Experience with CI/CD pipelines
• Knowledge of monitoring tools (Azure Monitor, Log Analytics, etc.)
• Exposure to DevOps or SRE practices
• Experience in regulated or large enterprise environments

Work Model

• 247 rotational shifts
• Hybrid / Onsite / Remote (as applicable)