We are seeking an experienced and motivated Senior Data Engineer with strong expertise in Python, PySpark, Azure Databricks, and Azure Data Factory (ADF). The ideal candidate will design and implement scalable data pipelines and play a key role in enabling data-driven decision-making by integrating data across various sources.
Key Responsibilities:
- Design, develop, and maintain ETL/ELT data pipelines using ADF, Databricks, and PySpark.
- Develop scalable and reusable data processing solutions in Azure cloud environment.
- Work with large-scale datasets from structured and unstructured data sources.
- Optimize and tune PySpark jobs for performance and reliability.
- Integrate and transform data across Azure services (Blob Storage, Data Lake, Synapse, etc.).
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
- Ensure data quality, security, and compliance throughout the data pipeline lifecycle.
- Develop and maintain documentation for data workflows and pipelines.
- Troubleshoot and resolve data pipeline issues efficiently.
Required Skills:
- 610 years of experience in data engineering or related field.
- Strong programming skills in Python.
- Extensive experience with PySpark for distributed data processing.
- Hands-on experience in Azure Databricks and ADF.
- Knowledge of Azure ecosystem (Data Lake, Blob Storage, Synapse, Key Vault, etc.).
- Good understanding of data warehousing and data modeling concepts.
- Experience working with CI/CD pipelines for data workflows.
- Strong problem-solving skills and ability to work independently.
- Excellent communication and stakeholder collaboration skills.
Preferred Qualifications:
- Microsoft Certified: Azure Data Engineer Associate or similar certifications.
- Experience with delta lake, streaming data using structured streaming.
- Exposure to DevOps, Terraform, or other IAC tools for Azure resource provisioning.
- Familiarity with data governance, data quality frameworks, and RLS.
