Lead Data Engineer

• Design and Develop Data Pipelines: Hands on with development, and optimisation of scalable and reusable - data pipelines in Azure Microsoft Fabric Synapse Data Engineering , leveraging both batch and real-time processing techniques . Ensure smooth integration with Azure Data Factory for orchestration and workflow management.
• Cloud Data Architecture: Collaborate with the Data Architecture team to design and implement robust data architectures in the Azure environment , ensuring they align with business needs while optimising performance, scalability, and cost-efficiency.
• Pipeline Optimisation : Continuously monitor and optimise the performance, cost, and reliability of data pipelines, ensuring efficient processing, storage, and management of large datasets.
• Cross-functional Collaboration: Work closely with data engineering teams , analysts , and business stakeholders to understand data requirements, developing solutions that enable self-service analytics and support the decision-making process.
• Documentation Knowledge Sharing: Contribute to internal documentation, fostering a culture of knowledge-sharing. Provide mentorship and guidance to junior engineers, helping to elevate team skills and improve overall team performance.
• Microsoft Fabric Experience: Apply your knowledge of Azure Tech Stack on Data Engineering ( or your willingness to learn ) Fabric-based development to manage end-to-end data orchestration, governance, and security across cloud and on-premises systems, ensuring seamless data movement and integration across hybrid environments.
• Data Modelling Expertise: Leverage your deep expertise in Azure to design and implement data models , create processing pipelines, and integrate with other Azure services like Data Lake and Synapse to support data storage and analytics needs.

Required Skills and Qualifications:

• Experience with Azure Ecosystem (Preferably Synapse): 5+ years of hands-on experience with Azure Ecosystem , including Synapse , Spark , OneLake , and other Fabric tools. Expertise in optimising Fabric notebooks and efficiently managing large-scale data workloads.
• Proficiency in Azure Data Factory: Strong experience with designing and orchestrating complex data pipelines using Azure Data Factory , with an emphasis on seamless data flow integration across various Azure services.
• Familiarity with Microsoft Fabric: A working knowledge or eagerness to learn Azure Data Fabric , focusing on cross-platform data orchestration, governance, and security.
• Advanced Data Engineering Skills: Extensive experience in data engineering, including the design and implementation of ETL processes and working with large datasets. Proven expertise in data quality , monitoring, and testing practices.
• Cloud Architecture Design Expertise: Experience designing and implementing data architectures in the Azure ecosystem , including tools such as Data Lake , Synapse , and Azure Storage .
• SQL and Data Modelling Expertise: Strong skills in SQL and data modelling , with the ability to design optimised data structures, tables, and views. Knowledge of both transactional and analytical data modelling.
• Collaboration and Communication Skills: Strong ability to work cross-functionally with teams from various domains. Ability to communicate complex technical concepts to both technical and non-technical stakeholders.
• Cost Optimisation: Proven experience optimising data engineering processes and Azure resources for both performance and cost, particularly in large-scale cloud environments.

Preferred Skills:

• Data Lakehouse Experience: Familiarity with Data Lakehouse architectures, particularly with tools like Delta Lake , OneLake , etc.
• Azure Ecosystem Familiarity: Knowledge of Azure s full ecosystem for end-to-end data integration and ETL processes.
• Proficiency in PySpark and Python: Expertise in PySpark for data processing tasks, with a solid foundation in Python .
• Fabric Integration: Familiarity with Fabric and how it integrates with other services within the Azure ecosystem .
• Databricks Experience: Experience with Databricks is a plus.