Azure Databricks Engineer

This role is for one of our clients

We are looking for a highly skilled Databricks Engineer to design, develop, and optimize data pipelines and analytics solutions. The ideal candidate will have strong expertise in Databricks, PySpark, and SQL, along with experience in creating scalable data solutions across cloud platforms.

Requirements

Key Responsibilities

• Design and develop data pipelines and ETL workflows using Databricks and PySpark.
• Create and enhance SQL queries for data transformation and analytical purposes.
• Work closely with data architects, analysts, and business stakeholders to deliver high-quality data solutions.
• Maintain data quality, governance, and compliance throughout all processes.
• Integrate data from diverse sources and effectively handle large-scale datasets.
• Diagnose issues and improve the performance of data workflows and queries.

Primary Skills (Must-Have)

• Strong proficiency in Databricks, including Delta Lake and machine learning capabilities.
• Advanced expertise in PySpark for distributed data processing.
• Excellent SQL skills for data manipulation and analytics.

Good-to-Have Skills

• Experience with cloud services on Azure (ADF, Synapse, Purview, Fabric).
• Knowledge of AWS services such as Glue, Lambda, and Step Functions.
• Familiarity with workflow orchestration tools like Airflow.
• Experience using data transformation and integration tools including DBT, Fivetran, and Informatica.
• Understanding of streaming and messaging platforms like Kafka.
• BI and data visualization skills with Power BI.
• Experience with data governance tools such as Collibra and Alation.
• Knowledge of Google Cloud Platform's BigQuery.

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related discipline.
• Demonstrated experience in constructing and managing data pipelines within cloud environments.
• Strong analytical and problem-solving abilities.
• Excellent communication and teamwork skills.

Nice to Have

• Experience operating in multi-cloud environments such as Azure, AWS, and GCP.
• Familiarity with modern data stack technologies and data governance frameworks.

Must-have skills

Databricks, PySpark, Microsoft Azure

Good-to-have skills

SQLDatabricks