Developing and deploying data engineering workloads in Databricks on the Azure cloud infrastructure, managing and monitoring performance of these workloads, and ensuring security, availability, and scalability of cloud resources
Specific Accountability
Work with the business lines to understand the data ingestion and cloud data engineering requirements
Work with the data modellers to translate the same into cloud/on-prem data engineering workloads design and then develop the same into appropriate engineering language Databricks pipelines and ADF pipelines.
Very strong hands-on skills and at least 3 years experience in developing databricks and Azure Data Factory pipelines
Follow the principle of medallion architecture (3 layers of data in Azure data lake) of getting the data in right data zones basis the requirement
Work and collaborate in multi-disciplinary Agile teams, adopting Agile spirit, methodology and tools
Help code and scale applications in a multi-cloud environment integrated with on-premises
Work with the platform lead to spot out and remediate the potential operational risks of the platform
