Experience: 5–8 Years
Location: Bangalore / Hyderabad / Chennai / Pune
We are looking for an experienced
Azure Data Engineer with strong expertise in
Python, PySpark, Azure Databricks, and Azure Cloud Services. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, processing large datasets, and supporting enterprise-grade data platforms. The role involves working closely with business stakeholders, building robust ETL solutions, and maintaining production data pipelines in a DevOps environment.
Key Responsibilities
- Design, develop, and optimize scalable data pipelines using Python, PySpark, and Azure Databricks.
- Build and maintain batch and streaming data processing solutions.
- Develop ETL/ELT workflows to ingest data from multiple structured and unstructured data sources.
- Work with business and technical teams to gather requirements and translate them into data engineering solutions.
- Develop data lake and data warehouse solutions using Azure services.
- Monitor, troubleshoot, and support production data pipelines.
- Collaborate with DevOps teams to automate deployments using CI/CD pipelines.
- Implement logging, monitoring, and performance tuning for data processing jobs.
- Ensure data quality, security, and governance best practices.
- Participate in code reviews, documentation, and technical design discussions.
Required Skills- 5+ years of experience in Data Engineering.
- Strong programming skills in Python.
- Hands-on experience with PySpark and Azure Databricks.
- Experience developing both Batch and Structured Streaming pipelines.
- Strong knowledge of SQL and exposure to NoSQL databases.
- Hands-on experience with Azure Cloud services, including:
- Azure Data Lake Storage (ADLS Gen2)
- Azure Databricks
- Azure Event Hub
- Azure Event Grid
- Azure SQL Data Warehouse / Azure Synapse Analytics
- Azure Functions
- Azure Key Vault
- Good understanding of Serverless Architecture.
- Knowledge of Infrastructure as Code using Terraform.
- Experience with Azure DevOps and CI/CD pipelines.
- Strong debugging, troubleshooting, and performance optimization skills.
Preferred Skills
- Azure Data Factory (ADF)
- ARM Templates
- Kubernetes (AKS)
- Google Cloud Platform (GCP)
- Machine Learning or AI exposure
- Experience with data lake and modern cloud data platform architectures
Preferred Candidate Profile
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
- Experience working in Agile/Scrum environments.
- Comfortable working in a DevOps model involving both development and production support.
- Ability to independently design and deliver scalable data engineering solutions.
Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
Skills: pyspark,azure,nosql,python
