Azure Databricks Data Engineer

Pradeepit Consulting Services is actively seeking an experienced Azure Databricks Data Engineer to join our client's dynamic team. This role demands a professional with expertise in Azure Databricks, PySpark, and SQL, focused on designing, developing, and optimizing data solutions.

Key Responsibilities

• ETL Development & Data Processing:
• Design, develop, and maintain ETL pipelines using Azure Databricks, PySpark, and Azure Data Factory (ADF).
• Extract, transform, and load data from multiple sources into Azure Data Lake Storage.
• Optimize and fine-tune ETL workflows for performance and scalability.
• SQL Query Optimization:
• Write complex SQL queries, including joins, subqueries, and functions, for data processing.
• Perform SQL tuning and query optimization to enhance efficiency.
• Efficiently handle large datasets.
• Azure Cloud Integration:
• Implement data solutions utilizing Azure technologies such as Databricks, ADF, and ADLS.
• Work with both batch and streaming data processing for real-time analytics (preferred).
• Ensure data security, compliance, and reliability in cloud-based systems.
• Collaboration & Problem-Solving:
• Collaborate with cross-functional teams to support data analytics and business intelligence initiatives.
• Proactively identify and resolve data-related issues.
• Provide insights and recommendations for improving ETL performance.

Skills

• Strong experience in Azure Databricks, PySpark, and SQL.
• Proficiency in writing complex SQL queries, functions, and performing query tuning.
• Hands-on experience with Azure Data Factory (ADF) and Azure Data Lake Storage.
• Ability to handle large datasets efficiently.
• Proven experience in performance optimization of ETL workflows.
• Good understanding of batch and streaming data processing (preferred).
• Strong analytical and problem-solving skills.
• Ability to work independently and take ownership of tasks.

Qualifications

• Bachelor's degree in Information Technology, Computer Science, or a related field.