Role Summary
We are seeking an
Azure Data Engineer / ADF Support Developer to support and enhance data pipelines within the
Risk Management & Analytics portfolio.
The role involves
monitoring daily pipeline executions, troubleshooting failures, managing API integrations, and ensuring smooth data ingestion and processing across environments. The ideal candidate will have strong expertise in
Azure Data Factory, API integrations, and operational support.
Key ResponsibilitiesPipeline Monitoring & Support
- Monitor Azure Data Factory (ADF) pipelines and scheduled jobs.
- Identify and troubleshoot pipeline failures, data ingestion errors, and API issues.
- Ensure smooth daily operations and timely issue resolution.
ADF Development & Enhancement
- Support and enhance ADF pipelines orchestrating API calls and data ingestion.
- Optimize pipelines for performance, reliability, and scalability.
- Maintain and update pipeline configurations as needed.
Azure Storage & Data Management
- Manage Azure Storage (Blob / Data Lake Gen2) for data ingestion and retention.
- Design and maintain folder structures, lifecycle policies, and data movement processes.
API Integration & Troubleshooting
- Handle REST APIs with OAuth 2.0 authentication.
- Troubleshoot issues related to API failures, authentication, connectivity, and data formats (JSON/XML).
Security & Key Management
- Manage secrets, credentials, and tokens using Azure Key Vault.
- Ensure secure access and perform credential rotation and environment sync.
Monitoring & Observability
- Monitor logs using ADF Monitor, Log Analytics, Application Insights.
- Identify issues proactively and escalate to vendors when required.
Environment & Vendor Coordination
- Maintain Dev, QA, and Prod environment configurations (keys, endpoints, parameters).
- Collaborate with vendors for issue resolution, validation, and deployments.
- Participate in weekly stakeholder discussions for continuous improvement.
Documentation & Collaboration
- Document data pipelines, workflows, and operational processes.
- Collaborate with Enterprise Architects (EA), Security teams (OIS), and business stakeholders.
- Train internal teams when required.
Required Skills Azure Data Engineering- Strong experience with Azure Data Factory (ADF):
- Pipelines, triggers, monitoring, debugging
- API orchestration and data ingestion
Cloud & Storage
- Hands-on experience with Azure Storage (Blob / Data Lake Gen2).
- Understanding of data lifecycle and storage management.
API & Integration
- Experience with REST APIs and OAuth 2.0 authentication.
- Strong knowledge of JSON/XML data formats and API troubleshooting.
Security
- Experience with Azure Key Vault for secrets and credential management.
Monitoring & Debugging
- Experience with Log Analytics, Application Insights, ADF monitoring tools.
- Strong troubleshooting and root cause analysis skills.
Database Skills
- Strong SQL knowledge.
- Experience with tools like DBeaver (nice to have).
Additional Skills
- Experience working with SaaS vendors and ticketing systems.
- Understanding of integration methodologies and cloud environments.
- Familiarity with Agile methodologies.
Soft Skills
- Strong communication and stakeholder management skills.
- Ability to work independently and handle production issues.
- Good documentation and training capability.
Skills: oauth 2.0 authentication,log analytics,azure monitor,troubleshooting & debugging,documentation skills,strong sql skills,dbeaver,vendor management,application insights,rest apis,json / xml,azure key vault,azure data factory,adf,azure storage
