PySpark

Teamware Solutions

Mumbai

2-4 Years

This job is no longer accepting applications

Posted 2 months ago

Job Description

Key Responsibilities:
PySpark Development:
Design, implement, and optimize PySpark solutions for large-scale data processing and analysis.
Develop data pipelines using Spark to handle data transformations, aggregations, and other complex operations efficiently.
Write and optimize Spark SQL queries for big data analytics and reporting.
Handle data extraction, transformation, and loading (ETL) processes from various sources into a unified data warehouse or data lake.
Data Pipeline Design & Optimization:
Build and maintain ETL pipelines using PySpark, ensuring high scalability and performance.
Implement batch and streaming processing to handle both real-time and historical data.
Optimize the performance of PySpark applications by applying best practices and techniques such as partitioning, caching, and broadcast joins.
Data Storage & Management:
Work with large datasets and integrate them into storage solutions such as HDFS, S3, Azure Blob Storage, or Google Cloud Storage.
Ensure efficient data storage, access, and retrieval through Spark and other tools (e.g., Parquet, ORC).
Maintain data quality, consistency, and integrity throughout the pipeline lifecycle.
Cloud Platforms & Big Data Frameworks:
Deploy Spark-based applications on cloud platforms such as AWS (Amazon EMR), Azure HDInsight, or Google Dataproc.
Work with cloud-native services such as AWS Lambda, S3, Google Cloud Storage, and Azure Data Lake to handle and process big data.
Leverage cloud data processing tools and frameworks to scale and optimize the PySpark jobs.
Collaboration & Integration:
Collaborate with cross-functional teams (data scientists, analysts, product managers) to understand business requirements and develop appropriate data solutions.
Integrate data from multiple sources and platforms (e.g., databases, external APIs, flat files) into a unified system.
Provide support for downstream applications and data consumers by ensuring timely and accurate delivery of data.
Performance Tuning & Troubleshooting:
Identify bottlenecks and optimize Spark jobs to improve performance.
Conduct performance tuning of both the cluster and individual Spark jobs, leveraging Spark's in-built tools for monitoring.
Troubleshoot and resolve issues related to data processing, application failures, and cluster resource utilization.
Documentation & Reporting:
Maintain clear and comprehensive documentation of data pipelines, architectures, and processes.
Create technical documentation to guide future enhancements and troubleshooting.
Provide regular updates on the status of ongoing projects and data processing tasks.
Continuous Improvement:
Stay up to date with the latest trends, technologies, and best practices in big data processing and PySpark.
Contribute to improving development processes, testing strategies, and code quality.
Share knowledge and provide mentoring to junior team members on PySpark best practices.
Required Qualifications:
2-4 years of professional experience working with PySpark and big data technologies.
Strong expertise in Python programming with a focus on data processing and manipulation.
Hands-on experience with Apache Spark, particularly with PySpark for distributed computing.
Proficiency in Spark SQL for data querying and transformation.
Familiarity with cloud platforms like AWS, Azure, or Google Cloud, and experience with cloud-native big data tools.
Knowledge of ETL processes and tools.
Experience with data storage technologies like HDFS, S3, or Google Cloud Storage.
Knowledge of data formats such as Parquet, ORC, Avro, or JSON.
Experience with distributed computing and cluster management.
Familiarity with Linux/Unix and command-line operations.
Strong problem-solving skills and ability to troubleshoot data processing issues.

More Info

Job Type:

Permanent Job

Industry:

Information Technology

Role:

Software Developer

Function:

IT/Software Development - Client Server

Employment Type:

Full time

About Company

Teamware Solutions

Teamware Solutions, a business division of Quantum Leap Consulting Private Limited, offers cutting edge industry solutions for deriving business value for our clients' staffing initiatives. Offering deep domain expertise in Banking, Financial Services and Insurance, Oil and Gas, Infrastructure, Manufacturing, Retail, Telecom and Healthcare industries, Teamware leads its service in offering skills augmentation and professional consulting services.

Job ID: 121753453

Jobs by Skill - IT

Jobs by Skill - Non IT

International Jobs

Last Updated: 24-03-2026 09:57:13 AM

Homejobs in MumbaiPySpark

Similar Jobs

Data Engineer | Python, PySpark, Cloud Data Platforms & Migration Expertise

Synechron

6-8 yrs

Mumbai, India

Skills:

Gitlab, Sql, SQL Server, DataFlow, Pyspark, Rest Apis, AWS, MySQL, Cloudformation, Oracle, Spark SQL, Azure, Gcp, Terraform, Jenkins, PostgreSQL, Web Services, Azure DevOps, Informatica Cloud Services IDMC IICS, Delta Lake, CI CD pipelines

Lead Software Engineer – Python, PySpark, AWS – Markets Regulatory

Chase Bank

5-7 yrs

Mumbai, India

Skills:

Spark SQL, Pyspark, Software Development Life Cycle, Terraform, Agile Methodologies, Python, Sql, Etl, Apache Spark Framework, Application Resiliency, Security