AWS/Azure Databricks Platform Admin

We are seeking an experienced Azure Databricks Platform Administrator to design operate secure and optimize enterprisescale Databricks environments on Microsoft Azure The role is responsible for platform governance workspace administration identity and access management networking security monitoring automation cost optimization and production support The ideal candidate combines strong Azure infrastructure knowledge with handson Databricks administration and works effectively with data engineering analytics security cloud and application teams

Key Responsibilities

• Administer Azure Databricks accounts workspaces clusters cluster policies pools SQL warehouses jobs notebooks repositories libraries and access controls across development test and production environments
• Manage workspace provisioning and configuration using Infrastructure as Code and automated deployment pipelines
• Implement identity and access management through Microsoft Entra ID single signon service principals managed identities groups rolebased access control and SCIM provisioning
• Administer Unity Catalog including metastores catalogs schemas tables volumes storage credentials external locations privileges and dataaccess governance
• Configure secure Azure networking including virtual network injection private endpoints Private Link network security groups route tables DNS firewalls and controlled egress
• Integrate Databricks with Azure Data Lake Storage Key Vault Azure DevOps or GitHub monitoring platforms and other approved enterprise services
• Establish cluster standards runtime policies library controls tagging autoscaling autotermination and workload isolation
• Monitor platform availability performance capacity audit logs job health and security events build dashboards and actionable alerts
• Lead incident problem change release and servicerequest management for the platform meeting agreed service levels and operational KPIs
• Troubleshoot complex workspace compute authentication authorization connectivity storage Spark and integration issues
• Optimize platform cost through usage analysis rightsizing policy enforcement commitment utilization chargebackshowback and removal of idle resources
• Plan and execute Databricks Runtime upgrades platform changes vulnerability remediation businesscontinuity testing and disasterrecovery activities
• Define and maintain platform standards runbooks architecture diagrams operating procedures support documentation and knowledge articles
• Partner with engineering architecture security compliance FinOps and vendor teams to onboard workloads and drive continuous improvement
• Mentor junior administrators and provide technical leadership for platform operations and governance