Cloud Architect

We are looking for a hands-on Azure Data Platform Architect with strong expertise in Databricks, DevOps (CI/CD), and IaC (Terraform/Bicep). This is a platform-focused architect role, not a data engineering or infrastructure admin role.

The ideal candidate must have strong hands-on expertise across Azure, Databricks, IaC, and DevOps, with the ability to design, automate, and optimize the data platform end-to-end. This is a platform-focused architectural role where the candidate acts as a Databricks Platform Owner, guiding teams, enforcing best practices, and driving automation, governance, and scalability across the environment.

The architect will own the Databricks platform setup, governance, automation, cost optimization, and best practices, while also enabling CI/CD and ADF monitoring.

Key Responsibilities

1. Databricks Platform Architecture

• Set up and configure Databricks workspaces, clusters, schemas, RBAC, and security.
• Define and enforce best practices:
• Cluster policies
• Workspace standards
• Cost control
• Job orchestration rules
• Automate Databricks (clusters, jobs, configurations) using Terraform/Bicep.
• Implement observability:
• Failed job alerts
• Cost and usage insights
• Performance reporting

2. ADF Monitoring & Automation

• Review and assess pipeline quality (not develop pipelines).
• Automate extraction of ADF metrics (failures, slow pipelines, cost insights).
• Configure integration runtimes and automate ADF operations when required.

3. DevOps (CI/CD) Enablement

• Implement CI/CD for Databricks and ADF using Azure Repos/GitHub + Azure DevOps.
• Ensure controlled promotion from DEV TEST PROD.
• Not responsible for full project management pipelinesonly deployment pipelines.

4. Infrastructure-as-Code (IaC)

• Use Terraform/Bicep to automate deployments of:
• Data product resource groups
• ADF
• Key Vault
• Databricks
• Basic networking components (deep expertise not required)

5. Data Platform Governance (Long Term)