Job Description: Azure Data Engineer
Experience Required: 5 6 Years
Role Overview
We are looking for an experienced
Azure Data Engineer to design, build, and support end-to-end data pipelines on the Azure ecosystem. The role involves extracting and transforming data from multiple source systems, building robust ADF/Synapse/Databricks pipelines, and supporting business-critical reporting and reconciliation processes.
Skills Required
- Azure Data Factory (ADF)
- Azure Databricks (ADB)
- Azure Active Directory
- Azure Storage
- Azure Data Lake Services (ADLS)
- Azure Key Vault
- Azure Logic Apps
- Azure SQL DB
- Azure HDInsight
- Good knowledge of Azure DevOps (ADO) services Repos, Boards, Build Pipelines (CI/CD)
- Good knowledge of developing data engineering frameworks and notebooks using Azure Databricks with Spark SQL, PySpark, and Python
- Good knowledge of Azure Synapse Analytics
Good-to-Have Skills
- GitHub
- PowerApps
- Azure Monitor and Log Analytics
- Power BI
- REST APIs
Daily Activities
- Extracting data from source systems and files in any format from SFTP locations.
- Processing the extracted data and storing it into staging tables in Azure Blob Storage.
- Transforming data based on the mapping (STTM) document to meet given requirements in ADB using SQL, Python, and PySpark code.
- Generating a final TLM file or report and loading it into the Smartstream environment.
- Building pipelines end-to-end using ADF and Synapse.
- Monitoring pipelines and alerts in ADF.
- Implementing new enhancements for the LBU.
- Analyzing and fixing issues raised by the business.
- Updating JIRA and documenting in Confluence.
Ideal Candidate Profile
- 56 years of hands-on experience in Azure data engineering, with strong exposure to ADF, Databricks, and Synapse.
- Solid programming skills in SQL, Python, and PySpark for building transformation logic and data engineering frameworks.
- Experience working with SFTP-based data extraction and file-based integrations.
- Familiarity with CI/CD practices using Azure DevOps.
- Strong analytical and issue-resolution skills, with experience supporting business/production issues.
- Good communication skills for documentation (Confluence) and task tracking (JIRA).
Azure Databricks
