Senior Data Engineer (PySpark, Databricks & ADF)
About The Opportunity
A fast-growing firm in the Data Engineering & Cloud Analytics sector delivering enterprise-grade data platforms and analytics solutions. We architect and operate scalable Lakehouse pipelines on Azure to drive reporting, ML, and real-time insights for global customers. This is an on-site role in India focused on building robust, production-grade ETL and data infrastructure using PySpark, Databricks and Azure Data Factory.
Role & Responsibilities
- Design and implement scalable batch and streaming ETL pipelines on Azure using PySpark and Databricks, ingesting data into Delta Lake.
- Author and maintain Azure Data Factory pipelines for orchestration, parameterization, scheduling, and operational runbooks.
- Tune and optimize Spark jobs for performance and costpartitioning, caching, shuffle optimization, and efficient file formats.
- Implement data quality, schema evolution, and lineage controls; build automated validation and reconciliation checks.
- Establish CI/CD for notebooks and infrastructure-as-code; manage source control, release pipelines, and environment promotion.
- Collaborate with data scientists, BI teams, and stakeholders to productionize models, expose curated datasets, and meet SLAs.
Skills & Qualifications
Must-Have
- PySpark
- Databricks
- Azure Data Factory
- Delta Lake
- Apache Spark
- Python
- SQL
- Git
Preferred
- Apache Airflow
- Azure DevOps
- Spark performance tuning
Qualifications
- Bachelor's degree in Computer Science, Engineering or equivalent preferred.
- Proven experience building production data pipelines on Azure/Databricks; ability to work on-site in India.
Benefits & Culture Highlights
- Work on high-impact, enterprise-scale data platforms and Lakehouse architectures.
- Collaborative engineering culture with opportunities to mentor and define best practices.
- Exposure to modern analytics and MLOps workflows with rapid career growth potential.
Location: India (On-site) This role requires hands-on presence at an India office to collaborate with engineering and analytics teams.
Skills: adfs,databricks,apache spark,python,sql,pyspark,azure
