Azure Databricks Engineer

Black And White Business Solutions is actively seeking a highly skilled Azure Databricks Engineer to join our growing data team. This pivotal role requires extensive experience in designing, building, and optimizing scalable and robust data pipelines using Azure Databricks, PySpark, and Spark SQL. The ideal candidate will be proficient in integrating diverse data sources, developing efficient ETL/ELT processes, and collaborating with cross-functional teams to deliver high-quality data products and solutions.

Key Responsibilities

• Design and build scalable and robust data pipelines leveraging Azure Databricks, PySpark, and Spark SQL for efficient data processing.
• Integrate data from various structured and unstructured data sources using Azure data services such as Azure Data Factory, Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.
• Develop and maintain comprehensive ETL/ELT processes for the ingestion, transformation, and storage of large datasets.
• Collaborate effectively with data scientists, data analysts, and other engineers to understand requirements and deliver high-quality data products and solutions that drive business value.
• Monitor, troubleshoot, and optimize existing data pipelines to ensure peak performance, reliability, and cost-efficiency.
• Ensure data quality, governance, and security compliance are rigorously upheld in all developed solutions, adhering to industry best practices and regulatory standards.
• Actively participate in architectural decisions and cloud data solutioning, contributing insights and expertise to shape our data platform strategy.

Required Skills

• Data Engineering Experience: 5+ years of experience in data engineering or related fields, demonstrating a strong foundation in data manipulation and processing.
• Azure Databricks & Apache Spark: Strong hands-on experience with Azure Databricks and deep expertise in Apache Spark for big data processing.
• Programming Proficiency: Proficiency in Python (PySpark), SQL, and proven experience with performance tuning techniques for data pipelines and queries.
• Azure Data Services: Extensive experience with core Azure data services including Azure Data Factory, Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.
• Data Concepts: Solid understanding of data modeling, data warehousing, and data lakes concepts and their application in enterprise environments.
• DevOps & Version Control: Familiarity with DevOps practices, CI/CD pipelines, and version control systems (e.g., Git) for collaborative development and automated deployments.

Qualification

• Any Graduate or Above

Experience

• 5 to 15 Years

Notice Period

• Only Immediate

Contact: SWETHA Black and White Business Solutions Pvt.Ltd. Bangalore, Karnataka, India. Contact Number: 8067432433 Email: [Confidential Information] | Website: www.blackwhite.in