AWS Databricks Admin

We are looking for a Senior Databricks Administrator with strong expertise in Databricks on AWS, focused on security, governance, and cost optimization. This role will own platform administration, IAM-based data access controls, Unity Catalog governance, and infrastructure automation. The ideal candidate should demonstrate deep AWS IAM knowledge, production support maturity, and strong stakeholder management skills.

Key Responsibilities

• Administer and manage Databricks workspaces (AWS E2 architecture)
• Design secure S3 access using IAM roles, trust policies, and instance profiles
• Implement fine-grained access control (FGAC) using Unity Catalog (catalogs, schemas, tables, system catalog)
• Manage data governance policies, storage credentials, and external locations
• Define and enforce cluster policies for security and cost control
• Monitor cluster utilization, implement autoscaling, and drive cost optimization (DBUs, Spot instances)
• Provision and manage Databricks infrastructure using Terraform or Asset Bundles (IaC)
• Troubleshoot production incidents related to IAM, storage, and clusters
• Build and maintain dashboards within Databricks for monitoring and insights
• Collaborate with security, infrastructure, and business teams

Required Skills

• 5+ years of Databricks administration on AWS
• Strong hands-on experience with AWS IAM (roles, trust relationships, external ID, STS, instance profiles)
• Good understanding of AWS networking basics (VPC, Route53)
• Strong experience with Unity Catalog and governance models (including system catalog)
• Experience with cluster policy management and cost optimization strategies
• Strong scripting skills in Python or Bash
• Experience with Infrastructure as Code (Terraform / Databricks Asset Bundles)
• Experience building dashboards in Databricks (SQL / Lakehouse monitoring)
• Proven experience supporting production environments and stakeholder communication

Preferred

• Experience with multi-workspace governance and centralized metastore
• Exposure to chargeback/showback models
• AWS Certification (Solutions Architect / Security)