Developer

About the Company

We are looking for a skilled PySpark Developer to design and implement scalable data pipelines on cloud platforms (AWS/Azure/GCP). The role involves working with big data technologies to process large datasets efficiently and support analytics and ML use cases.

About the Role

The role involves working with big data technologies to process large datasets efficiently and support analytics and ML use cases.

Responsibilities

• Develop and optimize data pipelines using PySpark
• Work on cloud platforms (AWS / Azure / GCP) for data engineering tasks
• Design ETL processes using Spark, Hive, and SQL
• Integrate data from multiple sources (APIs, databases, streaming systems)
• Work with data lakes and lakehouse architectures
• Optimize Spark jobs for performance and cost efficiency
• Collaborate with data scientists and analysts
• Implement CI/CD pipelines for data workflows

Qualifications

• Strong experience in PySpark / Apache Spark
• Cloud experience (any of the following):
• AWS: S3, EMR, Glue, Redshift
• Azure: Data Factory, Synapse, ADLS
• GCP: DataProc, BigQuery
• Proficiency in Python and SQL
• Experience with Hadoop ecosystem (Hive, HDFS)
• Knowledge of Airflow / orchestration tools
• Familiarity with Delta Lake / Iceberg / Hudi
• Understanding of data modeling and warehousing concepts

Required Skills

• Strong experience in PySpark / Apache Spark
• Cloud experience (any of the following): AWS, Azure, GCP
• Proficiency in Python and SQL
• Experience with Hadoop ecosystem (Hive, HDFS)

Preferred Skills

• Knowledge of Streaming (Kafka, Spark Streaming)
• DevOps exposure (Docker, Kubernetes)
• Experience with ML pipelines

Pay range and compensation package

Not specified in the job description.

Equal Opportunity Statement

We are committed to diversity and inclusivity in our hiring practices.