Azure Databricks Platform Admin

Responsibilities :

Azure Databricks Platform Administration Administer and manage Azure Databricks workspaces, clusters, and jobs Configure and maintain Databricks environments (Dev, QA, Prod) Monitor cluster performance, job execution, and system health Manage notebooks, libraries, and dependencies Perform upgrades, patches, and platform maintenance Cluster Management & Optimization Design and manage cluster configurations (auto-scaling, instance types, pools) Optimize job performance, resource utilization, and execution times Troubleshoot cluster failures, memory issues, and job bottlenecks Implement best practices for Spark performance tuning Security & Governance Implement and enforce data security and access controls (RBAC, ACLs, IAM) Manage workspace security, secrets (Key Vault integration), and credential passthrough Ensure compliance with enterprise security and governance policies Implement data governance, auditing, and lineage tracking Azure Cloud Integration Integrate Databricks with Azure services such as: Azure Data Lake Storage (ADLS Gen2) Azure Key Vault Azure Data Factory (ADF) Azure Synapse Analytics Manage connectivity and networking (VNet, Private Endpoints, NSGs)

Technical and Professional Requirements:

Core Skills 10-15 years of IT experience with strong Azure Databricks administration Hands-on expertise in Apache Spark and Databricks platform management Strong experience with Azure cloud services (ADLS, ADF, Key Vault, Networking) Proven experience in production support and platform operations Technical Skills Deep knowledge of cluster configuration, job scheduling, and performance tuning Strong understanding of Spark architecture and optimization techniques Experience with Terraform / ARM / Bicep (IaC tools) Knowledge of security models, IAM, RBAC, and data governance Familiarity with CI/CD tools and DevOps practices Preferred Skills Experience with Delta Lake and Lakehouse architecture Exposure to Python / PySpark / Scala Familiarity with monitoring tools (Azure Monitor, Splunk) Knowledge of multi-cloud environments (AWS/GCP) Experience in data engineering or ETL pipelines