Azure Data Engineer

Primary Skills: Databricks, Azure Data Factory, Azure Function App, PySpark, Python, SQL, GitHub Job Summary

We are seeking an experienced Data Engineer Lead with strong expertise in Azure cloud data engineering technologies, scalable ETL pipeline development, and modern big data processing frameworks. The ideal candidate will be responsible for designing, developing, optimizing, and leading enterprise-grade data engineering solutions using Databricks, Azure Data Factory, Azure Function Apps, PySpark, Python, and SQL.

The candidate should possess hands-on experience building scalable batch and real-time data pipelines, implementing cloud-native architectures, managing large datasets, and collaborating with cross-functional teams to deliver reliable and high-performance data solutions.

Key Responsibilities Data Pipeline Development

• Design, develop, and maintain scalable ETL/ELT pipelines using:
• Azure Databricks
• Azure Data Factory (ADF)
• PySpark
• Python
• Build efficient data ingestion frameworks for structured and unstructured data sources.
  

Azure Cloud Data Engineering

• Develop cloud-native data solutions on Microsoft Azure.
• Work extensively with:
• Azure Databricks
• Azure Function Apps
• Azure Data Lake
• Azure Blob Storage
• Azure SQL
• Azure Synapse Analytics
• Implement secure, scalable, and high-performance data architectures.
  

Big Data Processing

• Develop and optimize distributed data processing workflows using PySpark.
• Handle large-scale datasets with focus on:
• Performance optimization
• Cost efficiency
• Scalability
• Reliability
  

Data Transformation & Integration

• Develop complex SQL transformations and data models.
• Integrate data from multiple enterprise systems, APIs, and third-party sources.
• Ensure data quality, consistency, and governance standards.
  
  

Automation & CI/CD

• Manage code repositories and deployment workflows using GitHub.
• Implement CI/CD best practices for data engineering deployments.
• Automate operational workflows and monitoring processes.
  
  

Leadership & Collaboration

• Lead and mentor junior data engineers.
• Collaborate with:
• Architects
• Product teams
• Business analysts
• DevOps teams
• Stakeholders
• Participate in design discussions and technical decision-making.
  

Monitoring & Optimization

• Monitor pipeline performance and troubleshoot failures.
• Optimize Spark jobs, SQL queries, and data workflows.
• Implement logging, alerting, and operational monitoring mechanisms.
  
  

Strong Hands-on Experience With

Required Technical Skills Must-Have Skills Azure Data Engineering

• Azure Databricks
• Azure Data Factory (ADF)
• Azure Function Apps
• Azure Data Lake
• Azure Blob Storage
  
  

Programming & Processing

Strong Expertise In

• PySpark
• Python
• SQL
  
  

Data Engineering

Experience With

• ETL/ELT pipelines
• Batch processing
• Data transformation
• Data integration
• Data warehousing
  
  

Version Control & CI/CD

Hands-on Experience With

• GitHub
• Git workflows
• CI/CD pipelines
  
  

Cloud & Big Data

Experience Working With

• Distributed data processing
• Large-scale datasets
• Cloud-native architectures
  
  

Skills: azure,pyspark,python,sql