IT Consulting

Job Description:

• We are looking for an experienced Azure Databricks Developer Data Engineer to design build and optimize large scale data processing solutions on the Microsoft Azure cloud
• The ideal candidate will have strong hands on experience with Azure Databricks data engineering frameworks and modern data lake architectures to support analytics and business intelligence initiatives

Key Responsibilities:

• Design develop and optimize data pipelines using Azure Databricks PySpark Spark SQL
• Build and manage scalable data ingestion transformation and processing workflows
• Implement ETL ELT pipelines using Databricks Azure Data Factory and related tools
• Develop data solutions using Azure Data Lake Storage ADLS Gen2
• Optimize Spark jobs for performance cost and scalability
• Implement best practices for data modeling partitioning and schema management
• Work with structured semi structured and unstructured data
• Collaborate with business analytics and downstream BI teams
• Ensure data quality reliability and monitoring of data pipelines
• Support CI CD version control and automated deployments

Technical Requirements:

• Primary skills Technology Data On Cloud Platform Azure Data Lake ADL
• 5 9 years of overall experience in data engineering big data development
• 3 years of hands on experience with Azure Databricks
• Strong expertise in PySpark Spark SQL
• Experience with Microsoft Azure services including
• Azure Data Factory ADF
• Azure Data Lake Storage ADLS
• Azure SQL Synapse good to have
• Solid understanding of data lake lakehouse architectures
• Strong SQL skills for data transformation and analytics
• Experience with workflow orchestration and scheduling
• Knowledge of data warehousing and data modeling concepts
• Familiarity with Git and CI CD pipelines
• Excellent problem solving and communication skills

Additional Responsibilities:

• Databricks Certified Data Engineer Associate Professional
• Experience with Delta Lake
• Knowledge of streaming frameworks Structured Streaming Event Hubs
• Exposure to BI tools Power BI Tableau
• Experience working in Agile Scrum teams
• Basic understanding of Python best practices and libraries

Preferred Skills: