Data Support Engineer

KEY ACCOUNTABILITIES:

• The Azure Data Support Engineer focuses on data-related tasks in Azure.
• Manage, monitor, and ensure the security and privacy of data to satisfy business needs.
• Monitor real-time and batch processes to ensure data accuracy.
• Monitor Azure pipelines and troubleshoot where required.
• Enhance existing pipelines and Databricks notebooks as needed.
• Involved in the development stages of new pipelines as required.
• Troubleshoot pipelines, real-time replication jobs, and ensure minimum data lag.
• Available to work on a shift basis to cover monitoring during weekends (one weekend out of three).
• Act as an ambassador for DP World at all times when working; promoting and demonstrating positive behaviours in harmony with DP World's Principles, values, and culture. Ensure the highest level of safety is applied in all activities, understanding and following DP World's Code of Conduct and Ethics policies.
• Perform other related duties as assigned.

JOB CONTEXT:

• Responsible for monitoring and enhancing existing data pipelines using Microsoft Stack.
• Responsible for the enhancement of existing data platforms.
• Experience with Cloud Platforms such as Azure, AWS, Google Cloud, etc.
• Experience with Azure Data Lake, Azure Data Lake Analytics, Azure SQL Database, Azure Databricks, and Azure SQL Data Warehouse.
• Good understanding of Spark Architecture including Spark Core, Spark SQL, DataFrames, Spark Streaming, Driver Node, Worker Node, Stages, Executors, and Tasks.
• Good understanding of Big Data Hadoop and Yarn architecture along with various Hadoop Daemons such as Job Tracker, Task Tracker, Name Node, Data Node, Resource/Cluster Manager, and Kafka (distributed stream-processing).
• Experience in Database Design and development with Business Intelligence using SQL Server 2014/2016, Integration Services (SSIS), DTS Packages, SQL Server Analysis Services (SSAS), DAX, OLAP Cubes, Star Schema, and Snowflake Schema.
• Monitoring of pipelines in ADF and experience with Azure SQL, Blob storage, Azure SQL Data Warehouse.
• Experience in a support environment working with real-time data replication is a plus.

QUALIFICATIONS, EXPERIENCE AND SKILLS:

Qualification:

• Bachelor's/Master's in Computer Science/IT or equivalent.
• Azure certifications will be an added advantage (Certification in AZ-900 and/or AZ-204, AZ-303, AZ-304, AZ-400, DP200, DP201).
• ITIL certification is a plus.

Experience:

• 5 - 8 years.

Must Have Skills:

• Azure Data Lake, Data Factory, Azure Databricks.
• Azure SQL Database, Azure SQL Data Warehouse.
• Hadoop ecosystem.
• Azure Analytics services.
• Programming: Python, R, Spark SQL.

Good to Have Skills:

• MSBI (SSIS, SSAS, SSRS), Oracle, SQL, PL/SQL.
• Data Visualization, Power BI.
• Data Migration.