Data Platform Engineer

Project Role :Data Platform Engineer

Project Role Description :Assists with the data platform blueprint and design, encompassing the relevant data platform components. Collaborates with the Integration Architects and Data Architects to ensure cohesive integration between systems and data models.

Must have skills :Microsoft Azure Modern Data Platform

Minimum5year(s) of experience is required

Educational Qualification :15 years full time education

Job Description:

Development, customize and manage integration tools, databases, warehouses and analytical systems with the use of data related instruments/instances

Create and run complex queries and automation scripts for operational data processing.

Test the reliability and performance of each part of a system and cooperate with the testing team

Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.

Responsible for setting up tools to view data, generate reports, and create visuals

Monitoring the overall performance and stability of the system. Adjust and adapt the automated pipeline as data/models/requirements change.

Mentor and train colleagues where necessary by helping them learn and improve their skills, as well as innovate and iterate on best practices.

Solve complex issues with minimal supervision

Make improvement and process recommendations that have an impact on the business

Technology Stack Used & Required Knowledge:

Mastery of Azure Data Services with minimum of 4 years of such as Azure Data Factory, Azure Synapse, Azure Databricks, Azure SQL Database, Azure Cosmos DB, Azure Blob Storage/Data Lake Storage, and Azure Stream Analytics.

5+ years Experience in using of Python/ PySpark for data engineering.

Understanding of data types/ handling of different data models.

Excellent understanding of ETL cycle

Expertise in designing scalable and efficient data architectures on Azure is a plus

Understanding of descriptive and exploratory statistics, predictive modelling, evaluation metrics, decision trees, machine learning algorithms is a plus.

Good scripting and programming skills.

Expertise in implementing data governance frameworks and ensuring compliance using Azure Purview. Familiarity with Azure Data Catalog for managing and discovering data assets.

Understanding of Azure Key Vault for secure key management.

Azure scripting with PowerShell/Azure CLI

Others:

Flexibility in using different technologies/ platforms

Ability to shift gears quickly and cope with change

Analytical & Detail Oriented

Confident, proactive and self-motivated

Disciplined and Team first approach oriented