Azure Databricks Engineer

Azure Databricks Engineer

JD:

The Azure Databricks Engineer will be responsible for designing, developing, and implementing large-scale data processing solutions on the Azure Databricks platform. The role involves collaborating with data scientists, analysts, and business stakeholders to deliver robust, scalable, and high-performance analytics solutions that support business objectives.

Key Responsibilities

Develop and optimise data pipelines using Azure Databricks, Spark, and related technologies.

Design and implement ETL (Extract, Transform, Load) workflows to ingest data from diverse sources into Azure Data Lake and other storage solutions.

Collaborate with cross-functional teams to understand data requirements and deliver end-to-end solutions.

Monitor, troubleshoot, and enhance Databricks jobs for performance, reliability, and scalability.

Ensure data quality, security, and compliance with organisational and regulatory standards.

Document processes, technical designs, and best practices for ongoing maintenance and knowledge sharing.

Required Skills and Qualifications

Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

Proven experience working with Azure Databricks and Apache Spark.

Strong proficiency in Python, Scala, or SQL for data engineering tasks.

Familiarity with Azure cloud services such as Azure Data Lake, Azure Data Factory, and Azure Synapse Analytics.

Experience in developing and optimising ETL/ELT pipelines.

Excellent problem-solving, analytical, and communication skills.