Job Title: Senior Databricks Administrator – AWS
Role Overview
We are looking for a Senior Databricks Administrator with strong expertise in Databricks on AWS, focused on security, governance, and cost optimization. This role will own platform administration, IAM-based data access controls, Unity Catalog governance, and infrastructure automation. The ideal candidate should demonstrate deep AWS IAM knowledge, production support maturity, and strong stakeholder management skills.
Key Responsibilities
- Administer and manage Databricks workspaces (AWS E2 architecture)
- Design secure S3 access using IAM roles, trust policies, and instance profiles
- Implement fine-grained access control (FGAC) using Unity Catalog (catalogs, schemas, tables, system catalog)
- Manage data governance policies, storage credentials, and external locations
- Define and enforce cluster policies for security and cost control
- Monitor cluster utilization, implement autoscaling, and drive cost optimization (DBUs, Spot instances)
- Provision and manage Databricks infrastructure using Terraform or Asset Bundles (IaC)
- Troubleshoot production incidents related to IAM, storage, and clusters
- Build and maintain dashboards within Databricks for monitoring and insights
- Collaborate with security, infrastructure, and business teams
Required Skills
- 5+ years of Databricks administration on AWS
- Strong hands-on experience with AWS IAM (roles, trust relationships, external ID, STS, instance profiles)
- Good understanding of AWS networking basics (VPC, Route53)
- Strong experience with Unity Catalog and governance models (including system catalog)
- Experience with cluster policy management and cost optimization strategies
- Strong scripting skills in Python or Bash
- Experience with Infrastructure as Code (Terraform / Databricks Asset Bundles)
- Experience building dashboards in Databricks (SQL / Lakehouse monitoring)
- Proven experience supporting production environments and stakeholder communication
Preferred
- Experience with multi-workspace governance and centralized metastore
- Exposure to chargeback/showback models
- AWS Certification (Solutions Architect / Security)
