Job Title: Senior Azure Data Engineer
Experience: 48 Years
Location: Remote
Employment: Contract
We are seeking an experienced Azure Data Engineer to design, build, and optimize scalable data pipelines and analytics solutions on the Microsoft Azure platform. The ideal candidate will have strong hands-on expertise in Azure Data Factory (ADF), Azure Databricks, and Power BI, and will work closely with business stakeholders, data analysts, and data scientists to enable data-driven decision-making.
Responsibilities:
- Data Engineering & Pipeline Development
- Design, develop, and maintain end-to-end ETL/ELT pipelines using Azure Data Factory (ADF).
- Ingest data from multiple sources (databases, APIs, flat files, streaming sources) into Azure data platforms.
- Implement data transformations using Azure Databricks (PySpark / Spark SQL).
- Build and optimize data solutions using Azure Data Lake Storage (ADLS Gen2) and/or Azure Synapse Analytics.
- Ensure data quality, data validation, and schema enforcement.
- Implement partitioning, performance tuning, and cost optimization strategies.
- Collaborate with BI teams to model data for reporting and analytics.
- Support and optimize datasets consumed by Power BI dashboards and reports.
- Implement incremental refresh, row-level security (RLS), and performance best practices in Power BI.
- Automate workflows and scheduling using ADF triggers and pipelines.
- Monitor pipeline performance, failures, and data latency; troubleshoot and resolve issues.
- Implement data security, access control, and governance aligned with Azure best practices.
- Work closely with business analysts, data scientists, and application teams.
- Document data pipelines, architecture, and operational procedures.
- Participate in design reviews, code reviews, and continuous improvement initiatives.
Requirements:
- Strong experience with Azure Data Factory (ADF) pipelines, triggers, integration runtimes.
- Hands-on expertise in Azure Databricks with PySpark / Spark SQL.
- Experience building datasets and reports for Power BI.
- Solid understanding of data warehousing concepts (star schema, slowly changing dimensions).
- Proficiency in SQL for data analysis and transformations.
- Experience with Azure Data Lake Storage (ADLS Gen2).
- Experience with Azure Synapse Analytics.
- Knowledge of CI/CD pipelines using Azure DevOps or GitHub Actions.
- Exposure to Delta Lake, data versioning, and optimization techniques.
- Experience with streaming data (Event Hub / Kafka) is a plus.
- Familiarity with Python beyond Spark (for utilities, automation).
