Platform/ETL Engineer

Senior Consultant - Platform Engineer

Responsibilities

• Design, build, and optimize platform solutions in Azure Databricks to support scalable, high-performance, and secure data engineering workflows.

• Implement and maintain platform automation for infrastructure, monitoring, and deployment pipelines using modern DevOps practices.

• Align with best practices for platform architecture, governance, and security within Databricks.

• Collaborate with data engineers, modellers, and testers to ensure the platform meets technical and business requirements.

• Proactively monitor, troubleshoot, and resolve issues to maintain platform stability and performance.

Role Requirements

• Strong platform engineering experience with Azure Databricks and related Azure services.

• Familiarity with DevOps principles and CI/CD pipeline tools (e.g., Azure DevOps, Jenkins, GitHub Actions).

• Background in system Azure architecture, networking, Windows and Linux configuration, DevOps or SRE roles
• • Ability to design scalable solutions using Databricks features such as Delta Lake, Unity Catalog, and cluster policies.
• Write and deploy infrastructure as Code (IaC) /Configuration as Code (CaC) (Azure Resource Manager, Puppet, Ansible, Terraform or similar)
• • Strong problem-solving skills, with the ability to think strategically and avoid tactical or short-term solutions when possible.
• • Experience working in Agile/Scrum teams.

Experience/Skillset

• 8+ years of experience in platform engineering or a related field.

• Strong knowledge of Databricks, including Spark optimization, cluster management, and governance frameworks.

• Good knowledge of Azure DevOps based toolchains
• • Proficiency in Python, SQL, and PySpark, with the ability to implement robust and efficient solutions in Databricks.
• • Expertise in Azure services such as Azure Data Factory, Azure Storage, and Azure Key Vault.
• • Strong knowledge of SDLC and Agile methodologies.
• • Excellent verbal and written communication skills.
• • Proven ability to manage multiple projects and priorities effectively while delivering to tight deadlines.
•