Role Overview
This is a Senior Lead position focused on hands-on platform engineering, not an Architect role. The candidate will lead offshore implementation activities for the Databricks platform on GCP, including automation, configuration, governance, CI/CD pipeline development, FinOps support, monitoring, and operational documentation.
Key Responsibilities
- Execute Databricks platform implementation tasks as per the approved project backlog.
- Configure Databricks workspaces, Unity Catalog, schemas, storage credentials, external locations, and other platform components.
- Develop and maintain Terraform modules, automation scripts, and reusable deployment assets.
- Build and manage CI/CD pipelines for notebooks, workflows, jobs, SQL assets, libraries, and Databricks Asset Bundles.
- Implement Unity Catalog configurations, including access controls, permissions, lineage validation, and governance settings.
- Configure and support Databricks SQL Warehouses, dashboards, and query performance optimization.
- Implement cluster policies, job policies, SQL warehouse policies, logging, monitoring, and health checks.
- Support FinOps activities by validating resource tags, analyzing usage, preparing cost reports, and identifying optimization opportunities.
- Troubleshoot workspace, cluster, job, SQL warehouse, deployment, permission, and connectivity issues.
- Prepare deployment guides, SOPs, runbooks, configuration documents, and knowledge transfer materials.
- Collaborate with onshore leads, customer platform teams, and internal delivery teams during implementation and handover.
- Provide L2/L3 platform support during rollout and stabilization.
Required Skills & Experience
- 8+ years of experience in Data Platform Engineering, Cloud Engineering, DevOps, Analytics Engineering, or related roles.
- Strong hands-on experience with Databricks platform implementation and administration.
- Experience with:
- Unity Catalog
- Delta Lake
- Databricks Workflows
- Jobs & Clusters
- SQL Warehouses
- Permissions & Access Management
- Strong programming skills in Python, SQL, and scripting.
- Hands-on experience with Git, CI/CD pipelines, Databricks Asset Bundles, and deployment automation.
- Experience with Terraform or other Infrastructure as Code (IaC) tools.
- Good knowledge of Google Cloud Platform (GCP), including IAM, Service Accounts, GCS, Networking/VPC, and Monitoring.
- Understanding of platform governance, security, audit controls, and access management.
- Experience creating implementation guides, runbooks, operational documentation, and knowledge transfer materials.
- Ability to independently lead offshore execution while collaborating with onshore and customer teams.
Preferred Skills
- Databricks Data Engineer Associate/Professional certification (or equivalent practical experience).
- Experience with MLflow, MLOps, Model Registry, or Model Serving.
- Exposure to FinOps, cost optimization, usage reporting, and dashboarding.
- Knowledge of enterprise data governance, platform observability, security controls, and production support.
- Experience supporting post-implementation stabilization and steady-state platform operations.
