Hiring: Senior PySpark Lead / ETL Engineer
Location: Chennai
Experience: 10-12 Years
Work ode: 5 Days WFO
Notice Period: Imm to 15 Days (Apply those who can join only within max 15 Days NP)
We are hiring a Senior PySpark ETL Engineer to design, build, and own large‑scale, production-grade data pipelines using Apache Spark (PySpark) in enterprise data platforms.
Key Requirements
- 10–12 years of overall IT experience (strong Data Engineering / ETL focus)
- 3+ years of hands-on PySpark / Apache Spark (production)
- Strong experience in ETL / ELT pipeline design at scale
- Excellent SQL and data modeling skills
- Experience handling large datasets in distributed environments
- Strong pipeline ownership and production support mindset
Technical Skills
- PySpark, Spark SQL, performance tuning
- ETL/ELT patterns (incremental loads, SCD, deduplication)
- RDBMS (Postgres/MySQL), Data Lakes (Parquet/ORC)
- AWS EMR or Spark on Kubernetes
- Airflow / ADF / Databricks Workflows
Role Responsibilities
- Build and own end-to-end PySpark ETL pipelines
- Optimize Spark jobs for performance and cost
- Handle production issues, SLAs, and root cause analysis
- Implement data quality, reconciliation, and governance
- Review code and mentor junior engineers
