About The Opportunity
A fast-scaling technology services firm operating in the Data Engineering and Cloud Analytics space, we partner with enterprises to build scalable data pipelines, automate ETL workflows, and unlock real-time insights from structured and unstructured datasets. Our Python PySpark developers architect and deploy high-performance data solutions on cloud platforms—enabling faster decision-making, predictive modeling, and regulatory compliance for global clients.
Role & Responsibilities
- Design, develop, and optimize PySpark ETL pipelines for large-scale batch and streaming data processing.
- Collaborate with data scientists and analysts to ingest, transform, and model data for ML training and reporting use cases.
- Implement data quality checks, error handling, and logging frameworks within Spark jobs for production-grade reliability.
- Integrate PySpark workflows with cloud data lakes (AWS S3, Azure Data Lake), metastores (Hive, Glue), and orchestration tools (Airflow, Luigi).
- Performance-tune Spark applications—partitioning, caching, broadcasting, and resource allocation—to reduce job runtimes and costs.
- Write clean, modular, testable Python code with unit/integration tests and document architecture decisions for team scalability.
Skills & Qualifications
Must-Have
- PySpark
- Python
- Apache Spark
- ETL Development
- SQL
- Data Lake Architecture
- AWS S3
- Apache Airflow
Preferred
- Azure Data Lake
- Databricks
- Delta Lake
Benefits & Culture Highlights
- On-site collaborative workspace in major Indian tech hubs with modern infrastructure and R&D labs.
- Opportunities to upskill in cloud-native data platforms and GenAI-powered data pipelines.
- Fast-track career growth with cross-functional exposure to data science, ML engineering, and cloud architecture teams.
Skills: data,python,lake,spark,cloud
