Role Summary

We are seeking an Azure Data Engineer / ADF Support Developer to support and enhance data pipelines within the Risk Management & Analytics portfolio.

The role involves monitoring daily pipeline executions, troubleshooting failures, managing API integrations, and ensuring smooth data ingestion and processing across environments. The ideal candidate will have strong expertise in Azure Data Factory, API integrations, and operational support.

Key ResponsibilitiesPipeline Monitoring & Support

Monitor Azure Data Factory (ADF) pipelines and scheduled jobs.
Identify and troubleshoot pipeline failures, data ingestion errors, and API issues.
Ensure smooth daily operations and timely issue resolution.

ADF Development & Enhancement

Support and enhance ADF pipelines orchestrating API calls and data ingestion.
Optimize pipelines for performance, reliability, and scalability.
Maintain and update pipeline configurations as needed.

Azure Storage & Data Management

Manage Azure Storage (Blob / Data Lake Gen2) for data ingestion and retention.
Design and maintain folder structures, lifecycle policies, and data movement processes.

API Integration & Troubleshooting

Handle REST APIs with OAuth 2.0 authentication.
Troubleshoot issues related to API failures, authentication, connectivity, and data formats (JSON/XML).

Security & Key Management

Manage secrets, credentials, and tokens using Azure Key Vault.
Ensure secure access and perform credential rotation and environment sync.

Monitoring & Observability

Monitor logs using ADF Monitor, Log Analytics, Application Insights.
Identify issues proactively and escalate to vendors when required.

Environment & Vendor Coordination

Maintain Dev, QA, and Prod environment configurations (keys, endpoints, parameters).
Collaborate with vendors for issue resolution, validation, and deployments.
Participate in weekly stakeholder discussions for continuous improvement.

Documentation & Collaboration

Document data pipelines, workflows, and operational processes.
Collaborate with Enterprise Architects (EA), Security teams (OIS), and business stakeholders.
Train internal teams when required.

Required Skills Azure Data Engineering

Strong experience with Azure Data Factory (ADF):

Pipelines, triggers, monitoring, debugging
API orchestration and data ingestion

Cloud & Storage

Hands-on experience with Azure Storage (Blob / Data Lake Gen2).
Understanding of data lifecycle and storage management.

API & Integration

Experience with REST APIs and OAuth 2.0 authentication.
Strong knowledge of JSON/XML data formats and API troubleshooting.

Security

Experience with Azure Key Vault for secrets and credential management.

Monitoring & Debugging

Experience with Log Analytics, Application Insights, ADF monitoring tools.
Strong troubleshooting and root cause analysis skills.

Database Skills

Strong SQL knowledge.
Experience with tools like DBeaver (nice to have).

Additional Skills

Experience working with SaaS vendors and ticketing systems.
Understanding of integration methodologies and cloud environments.
Familiarity with Agile methodologies.

Soft Skills

Strong communication and stakeholder management skills.
Ability to work independently and handle production issues.
Good documentation and training capability.

Skills: oauth 2.0 authentication,log analytics,azure monitor,troubleshooting & debugging,documentation skills,strong sql skills,dbeaver,vendor management,application insights,rest apis,json / xml,azure key vault,azure data factory,adf,azure storage

Azure Data Engineer IRC291482

Azure Data Engineer

Azure Data Engineer

Azure Data Engineer

Azure Data Engineer