Job Title: Databricks Administrator
Experience Required: 7 to 15 Years
Work Location: Bangalore / Chennai / Hyderabad/ Pune/ Gurgaon/ Jaipur
Role Overview:
We are seeking an experienced Databricks Administrator to manage, optimize, and maintain our Databricks platform across multiple environments. The ideal candidate will have a strong background in data engineering infrastructure, cluster management, security, governance, and automation within Databricks and related Azure/AWS ecosystems.
Key Responsibilities:
- Databricks Platform Administration:
- Manage and maintain Databricks workspaces, clusters, jobs, pools, and libraries.
- Monitor performance and optimize resource utilization across environments (dev, test, prod).
- Implement autoscaling, cluster policies, and cost control measures.
- Security & Access Management:
- Configure and maintain user/group access via Azure AD or IAM integration.
- Implement fine-grained access control using Unity Catalog and role-based permissions.
- Ensure compliance with enterprise data security standards.
- Automation & CI/CD:
- Automate cluster provisioning, job deployment, and workspace management using Terraform, Databricks CLI, or REST APIs.
- Integrate Databricks with CI/CD tools like Azure DevOps, Jenkins, or GitHub Actions.
- Monitoring & Troubleshooting:
- Proactively monitor cluster health, job performance, and data pipeline stability.
- Diagnose and resolve system, configuration, and performance issues.
- Implement alerting, logging, and audit trail monitoring.
- Integration & Collaboration:
- Work with Data Engineers, DevOps, and Cloud teams to ensure smooth data operations.
- Support integration with Snowflake, Kafka, Azure Data Lake, or AWS S3.
- Collaborate on optimizing Spark configurations and ETL performance.
- Governance & Compliance:
- Implement best practices for workspace and notebook governance.
- Manage version control for notebooks and data pipelines.
- Enforce tagging, cost allocation, and audit standards.
Required Skills & Qualifications:
- Bachelor's degree in Computer Science, Engineering, or related field.
- 7+ years of experience in Data Engineering or Platform Administration, with at least 3+ years in Databricks Administration.
- Strong understanding of Apache Spark architecture and performance tuning.
- Hands-on experience with Azure Databricks or AWS Databricks environments.
- Proficiency with Terraform, Databricks CLI, REST APIs, and CI/CD pipelines.
- Expertise in cluster configuration, job scheduling, and resource optimization.
- Familiarity with Unity Catalog, Delta Lake, DBFS, and Lakehouse architecture.
- Experience integrating Databricks with ADLS, S3, Snowflake, Synapse, or Redshift.
- Strong problem-solving and scripting skills (Python, Shell, or PowerShell).
- Excellent communication and documentation abilities.
