Lead-Data Bricks

Key Responsibilities

Data Pipeline Development

• Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.

• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.

• Develop and maintain data processing workflows using Delta Lake architecture.

• Ensure efficient data ingestion, transformation, and loading processes.

Data Management & Optimisation

• Optimise performance of data pipelines and processing workloads.

• Ensure data quality, reliability, and consistency across data platforms.

• Write and optimise complex SQL queries for data processing and validation.

• Implement data governance and access controls.

Cloud & Data Platform Integration

• Work with Azure Data Lake for storing and managing large datasets.

• Manage data cataloging and governance using Unity Catalog.

• Integrate data workflows with CI/CD pipelines for automated deployments.

• Support scalable and secure cloud-based data architectures.

Collaboration & Migration Projects

• Work closely with cross-functional teams to deliver data-driven solutions.

• Participate in data migration and modernisation initiatives.

• Support troubleshooting and resolution of data pipeline issues.

• Document data architecture, pipelines, and processes.

Required Skills & Experience

• Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks.
• Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.
• Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL).
• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Optimise performance of data processing and ensure data quality standards.
• Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.
• Experience with orchestration tools like Airflow or Azure Data Factory is a plus.
• Have working experience on a migration project.

Relevant Experience Preferred

• Hands-on experience developing data pipelines using Azure Databricks and PySpark.

• Experience implementing ETL/ELT solutions for enterprise data platforms.

• Experience optimizing data processing performance and data workflows.

• Experience working with Azure Data Lake and cloud-based data architectures.

• Experience working on data migration or modernization projects.

• Experience integrating data pipelines with CI/CD workflows

Qualification :

Any degree equivalent to IT or relevant