Senior Cloud Engineer (Azure/Databricks)

ABOUT THE ROLE

We are seeking a Senior Cloud Engineer to design, implement, and administer the Azure

infrastructure behind an enterprise data and AI platform — centered on Azure Databricks, ADLS

Gen2, Azure Data Factory, and Azure Machine Learning. This person will own the platform

foundation (infrastructure as code, network and identity architecture, compute governance,

observability, and cost management) and partner closely with Databricks administration, data

engineering, and data science teams.

This role suits an engineer who enjoys being the technical reference point for a platform — setting

architectural direction, partnering with adjacent teams on complex problems, and translating

business requirements into platform capabilities.

KEY RESPONSIBILITIES

• Infrastructure as Code (Terraform-first): Author and maintain reusable Terraform modules

provisioning the Azure data estate — Databricks workspaces, ADLS Gen2, Data Factory, Key

Vault, networking, and Azure ML; own remote state, module versioning, and drift detection.

• Network & Security Architecture: Design private networking and security posture — private

endpoints, hub-and-spoke topology, VNet injection, NSGs/firewall rules, managed identities,

Key Vault-backed secrets, RBAC, and data exfiltration controls.

• Data Lake Infrastructure: Own ADLS Gen2 at the infrastructure layer — storage architecture,

hierarchical namespace, ACL strategy, lifecycle/tiering, encryption, and access patterns.

• Observability & Reliability: Build and maintain the monitoring layer — Azure Monitor, Log

Analytics, diagnostics, alerting, and operational runbooks.

• FinOps & Capacity Planning: Own cost visibility and optimization across DBU and storage spend

— tagging, chargeback/showback, budget alerts, and capacity planning.

• CI/CD & Environment Management: Own deployment pipelines and the promotion path across

dev, test, and production environments.

• Databricks Account & Workspace Administration: Account console configuration, workspace

provisioning and topology, admin role delegation, and multi-workspace strategy (day-to-day

Databricks operations sit with a dedicated admin team; this role sets the standards they work

from).

• Unity Catalog: Metastore design, catalog/schema/table permission models, storage credentials,

lineage/audit, and Delta Sharing configuration.

• Identity & Access: Entra ID integration, SCIM provisioning, identity federation, service

principals, and token policy.

• Compute Governance: Cluster policies, instance pools, autoscaling/autotermination standards,

and SQL warehouse sizing.

• Cost & Usage Analysis: System tables, usage attribution, and DBU forecasting, with the ability

to recommend remediation for cost spend.

• Technical Partnership: Publish standards, reference configurations, and self-service patterns;

support data teams on cross-cutting issues.

• Cross-Boundary Diagnostics: Investigate job failures, cluster/connectivity/permission issues,

and isolate root cause across infrastructure, Databricks, or workload.

ROLE SCOPE

• Production ETL/ELT pipeline development and Spark transformation work — owned by the Data

Engineering team.

• Model development, training, and tuning — owned by the Data Science team.

• Dimensional modeling, dbt development, and BI/semantic layer work — outside this role.

REQUIRED QUALIFICATIONS (MANDATORY)

• Terraform: Expert-level, hands-on. Authored and maintained production Terraform modules,

managed remote state, and run IaC through CI/CD — a core daily skill.

• Databricks Administration: Demonstrable hands-on experience — account/workspace

administration, Unity Catalog, cluster policies, identity federation, and cost governance.

• Technical Partnership: Proven track record as senior technical resource to adjacent teams,

guiding stakeholders from request to requirement and building consensus.

• Azure Networking & Security: Deep knowledge of private endpoints, VNets/hub-spoke design,

NSGs, Entra ID, managed identities, RBAC, and Key Vault.

• Data Lake Architecture: ADLS Gen2 design and access control at enterprise scale.

• Azure Data Factory: Platform-side experience with integration runtimes, managed VNet,

credential management, and deployment automation.

• Automation & Scripting: Strong Python, plus PowerShell and/or Bash, for platform tooling and

automation.

• Spark & SQL Literacy: Sufficient working knowledge to diagnose infrastructure/configuration-

level performance issues; deep Spark development is not required.

PREFERRED QUALIFICATIONS

• Databricks Asset Bundles, the Terraform Databricks provider, and workspace-as-code patterns.

• Experience leading a Unity Catalog migration or multi-workspace consolidation.

• Azure Machine Learning, MLflow, or model-serving infrastructure experience.

• Kubernetes / AKS and containerized workloads.

• Policy as code — Azure Policy, OPA, Sentinel, or Checkov.

• Event-driven infrastructure — Event Hubs, Kafka, or Stream Analytics.

• Formal FinOps practice experience.

• Multi-region or multi-tenant Databricks deployments.

PREFERRED CERTIFICATIONS

• Databricks Certified Data Engineer Professional, or a Databricks platform administrator

credential.

• Microsoft Certified: Azure Solutions Architect Expert (AZ-305).

• Microsoft Certified: Azure Administrator Associate (AZ-104).

• Microsoft Certified: DevOps Engineer Expert (AZ-400).

• HashiCorp Certified: Terraform Associate.

If interested please share your resume to [Confidential Information]