. Defines and implements solution for Data Analytics to meet business needs including data integration & pipeline needs and interfaces.
. Responsible for implementing robust data pipeline using Microsoft Stack, creating reusable and scalable data pipelines
. Responsible for development and deployment of new data platforms
. Create pipelines in ADF using linked services/Data sets to extract, Transform and load data from different sources like Azure SQL, Blob storage, etc.
. Develop spark applications using the Py spark and Spark-SQL for data extractions, transformation & Aggregation from multiple file formats.
. Responsible for estimating the cluster , monitoring and troubleshooting of the Data bricks cluster.
. Develop json scripts for deploying the pipeline in Azure Data Factory (ADF) that process the data using SQL activity.
. Collaborating with Azure Cloud Solution Architects and Data Platform Engineers in developing complex end-to-end Enterprise solutions on Microsoft Azure platform.
. Maintains health and procedures on data pipelines for continued success
. Involve in technical design of solutions working collaboratively with development leads, Architecture teams and development/implementation of solutions on Data Engineering/Analysis.
. Develop initial guidelines and support structures to ensure the service and assets are properly managed and maintained
. Collaborating with Team members in developing applicable methodologies, tools, approaches, point of view, thought leadership and accelerators to differentiate EY in the Digital & Emerging Technology Advisory Market
. Develop reusable assets such as prototypes, solutions, documentation and other materials that can help grow the practice
. Design & build API & data integrations between the various applications in the Platform utilizing modern cloud based stack like Azure Service Bus, Azure Functions and Azure Data Factory.
What can help make you successful in this role
Bachelors or Master's degree in Computer Science or related field or have relevant experience.
5-10 years of relevant experience as a Senior Data Engineer with developing pipelines using Azure Data Factory, Functions
5-10 years of relevant experience in Python, Spark, SQL, Javascript/Powershell
5 -10 years of Deep understanding on Data Analysis, schema design and dimensional data modeling, Data lake/Data Bricks technology is a big plus
2+ years of Hands on experience developing Data Pipelines (ideally loading Databases using ETL approaches) using SQL & no SQL data sources and Enterprise data pipeline tools like Azure Data factory, Data Bricks or similar technologies.
3+ Years of experience in understanding of Data import/export approaches, tooling, and data quality issues.
2+ years of experience on cloud platforms (AWS, Azure, or GCP)
5+ years experience with latest Microsoft software delivery and collaboration stack (Visual Studio / Azure DevOps, SharePoint, Teams, Office 365, etc.)
years of Experience working in an Agile environment to facilitate the quick and effective fulfillment of group goals
Knowledge of various industry-leading backend databases and queuing technologies (relational/NoSQL databases, SQL functions, queries)
Self-directed, works with limited management direction, and exercises considerable latitude in determining technical objectives of assignment
Ability to prioritize workload according to volume, urgency, etc. and to deliver on required projects in a timely fashion
Passion for learning & immersing in other technology stack like Python, Spark etc. to be part of versatile Development team
Communication skills demonstrated by winning stakeholders and communicating with team members
Multitask experience against multiple work streams following an End-to-End workflow process
What makes you stand out:
Strong understanding & documented hands on experience in Azure Cloud, Azure Data Lake, Altryx including Building clusters, Building pipelines & CI/CD.
Familiarity with modern Cloud native Spark based platforms like Amazon EMR or Databricks
Strong knowledge of CI/CD principles and practical experience with a CI/CD technology (Azure Devops, GitLab, Jenkins).
Insurance or Reinsurance industry experience.
