Job Summary
We are seeking an experienced
Senior Azure Data Engineer to design, develop, and support scalable data ingestion and processing solutions within an Azure Databricks environment. The role involves building robust ETL/data pipeline solutions, integrating enterprise data sources, enabling data services for .NET applications, and supporting API-based data publishing workflows.
The ideal candidate should possess strong expertise in
Azure Data Factory, Azure Databricks, Azure Synapse, Spark, Python, SQL, and cloud-native data engineering practices with hands-on experience managing large-scale structured and unstructured datasets.
Key Responsibilities
- Design, build, and maintain scalable data pipelines for processing large volumes of enterprise data.
- Develop and optimize ETL/ELT workflows using Azure Data Factory (ADF), Databricks, and Synapse.
- Implement data ingestion, transformation, cleansing, and validation processes.
- Build and maintain data services/APIs supporting .NET-based business applications.
- Develop and optimize SQL queries and Spark-based transformations for high-performance data processing.
- Monitor, troubleshoot, and optimize data pipelines, ETL jobs, and performance bottlenecks.
- Implement data governance, metadata management, lineage tracking, and data quality processes.
- Work closely with data scientists, analysts, developers, and publishing teams to deliver scalable data solutions.
- Handle structured and unstructured datasets from multiple enterprise sources.
- Participate in defect tracking, root cause analysis, and production support activities.
- Ensure compliance with organizational data security and privacy standards.
- Stay current with emerging technologies and best practices in cloud data engineering.
Required Skills & Experience- 7+ years of experience in Data Engineering and Azure cloud technologies.
- Strong hands-on experience with:
- Azure Data Factory (ADF)
- Azure Databricks
- Azure Synapse Analytics
- Azure SQL Database
- Azure Functions
- Azure Logic Apps
- Strong experience building:
- ETL/ELT pipelines
- ADF Data Flows & Activities
- Spark/PySpark transformations
- Expertise in:
- Experience working with:
- Structured & unstructured data
- Data quality and validation frameworks
- Data governance and metadata management
- Strong understanding of:
- Data Lake architecture
- Distributed data processing
- API-based data integration
- Experience with technologies such as:
- Snowflake
- Dremio
- Kafka
- Hadoop
- Excellent analytical, troubleshooting, and communication skills.
- Strong collaboration skills with technical and business stakeholders.
Preferred Skills
- Experience extracting data from SharePoint Online.
- Familiarity with Dremio data virtualization platform.
- Exposure to enterprise data governance initiatives.
- Experience supporting .NET application integrations.
- Knowledge of DevOps and CI/CD practices in Azure.
Certifications (Preferred)
- DP-203: Data Engineering on Microsoft Azure
- Azure Data Engineer Associate
- Databricks Certifications
Skills: oauth 2.0 authentication,application insights,azure monitor,log analytics,data engineering,vendor management,json / xml,troubleshooting & debugging,rest apis,azure databricks,documentation skills,azure key vault,adf,strong sql skills,azure data factory,dbeaver,azure storage
