Pyspark

Responsibilities :

Key Responsibilities:

Design, develop, and maintain scalable ETL/ELT pipelines using PySpark for batch and/or incremental processing.

Build and optimize Apache Spark jobs with focus on performance, partitioning strategy, caching, and efficient transformations/actions.

Perform data cleansing, validation, and reconciliation to ensure accuracy, completeness, and consistency of datasets.

Collaborate with cross-functional teams to understand requirements and translate them into robust data processing solutions.

Troubleshoot pipeline failures, analyze logs, identify bottlenecks, and implement fixes to improve reliability and throughput.

Write clean, maintainable code with reusable components and clear documentation for pipelines and data flows.

Support deployment and operationalization of Spark workloads, including monitoring and basic production support activities.

Contribute to code reviews and follow engineering best practices to improve quality and maintainability. Minimum Qualifications:

Education: BTECH, MTECH, MCA, MSC.

2â€3 years of experience in data engineering or large-scale data processing roles.

Strong hands-on experience with PySpark for building data pipelines and transformations.

Working knowledge of Apache Spark concepts such as RDD/DataFrame, joins, shuffles, and performance considerations.

Ability to debug Spark applications and resolve data/job issues effectively.

Additional Responsibilities:

Preferred Qualifications:

Experience optimizing Spark workloads (tuning partitions, managing skew, memory/executor settings) for performance and cost efficiency.

Exposure to building end-to-end data pipelines with strong data quality checks and automated validations.

Familiarity with distributed processing patterns and designing reusable PySpark modules for scalable development.

Experience collaborating in agile teams, participating in code reviews, and improving engineering standards for data pipelines.

Technical and Professional Requirements:

Good to have skills: SQL, Hadoop, Hive, Kafka, Airflow