Pyspark

Good to have skills: SQL, Hadoop, Hive, Kafka, Airflow

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark for batch and/or incremental processing.
• Build and optimize Apache Spark jobs with focus on performance, partitioning strategy, caching, and efficient transformations/actions.
• Perform data cleansing, validation, and reconciliation to ensure accuracy, completeness, and consistency of datasets.
• Collaborate with cross-functional teams to understand requirements and translate them into robust data processing solutions.
• Troubleshoot pipeline failures, analyze logs, identify bottlenecks, and implement fixes to improve reliability and throughput.
• Write clean, maintainable code with reusable components and clear documentation for pipelines and data flows.
• Support deployment and operationalization of Spark workloads, including monitoring and basic production support activities.
• Contribute to code reviews and follow engineering best practices to improve quality and maintainability. Minimum Qualifications:
• Education: BTECH, MTECH, MCA, MSC.
• 2–3 years of experience in data engineering or large-scale data processing roles.
• Strong hands-on experience with PySpark for building data pipelines and transformations.
• Working knowledge of Apache Spark concepts such as RDD/DataFrame, joins, shuffles, and performance considerations.
• Ability to debug Spark applications and resolve data/job issues effectively. Preferred Qualifications:
• Experience optimizing Spark workloads (tuning partitions, managing skew, memory/executor settings) for performance and cost efficiency.
• Exposure to building end-to-end data pipelines with strong data quality checks and automated validations.
• Familiarity with distributed processing patterns and designing reusable PySpark modules for scalable development.
• Experience collaborating in agile teams, participating in code reviews, and improving engineering standards for data pipelines.