We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and architectures, databases. The ideal candidate will have hands-on experience in Azure Data Factory (ADF), Azure Synapse, and SQL, and will support reporting teams by ensuring reliable, curated datasets are available for downstream analysis.
Key Responsibilities:
- Design, develop, and manage scalable data pipelines to ingest and transform data from multiple source systems using Azure Data Factory (ADF) and Synapse Analytics.
- Build and maintain data integration workflows that enable downstream analytics, ensuring data is clean, consistent, and timely.
- Partner with power bi developer and analysts within team and cross functional to develop data models optimized for reporting and machine learning.
- Implement data validation and quality checks to detect anomalies or pipeline failures early, ensuring high data reliability.
- Maintain and improve existing ETL frameworks, enabling incremental data loads, performance tuning, and reusability of components.
- Collaborate with the Power BI development team to ensure data availability in curated formats that support self-service analytics.
- Write and maintain advanced SQL scripts, stored procedures, and views to transform and manipulate large datasets efficiently.
- Monitor pipeline performance and implement logging, alerting, and retry mechanisms to handle failures.
- Stay updated on Azure tools and recommend newer approaches (e.g., Dataflows, Delta Lakes) where beneficial.
