Pyspark Data Engineer

Walk in Kolkata TCS Delta park 8 TH august 2026

Experience range 6+ years

Job Summary

We are seeking an experienced PySpark Data Engineer with 6+ years of experience in Data Engineering, Big Data, and cloud-based data platforms. The ideal candidate should have strong expertise in PySpark, Apache Spark, Python, SQL, Databricks, and Cloud Technologies to design, develop, and optimize scalable data processing solutions.

The candidate will be responsible for developing enterprise-grade data pipelines, implementing ETL/ELT frameworks, processing large-scale datasets, and supporting modern data lakehouse architectures.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using PySpark and Apache Spark.

Build and optimize ETL/ELT workflows for processing structured, semi-structured, and unstructured data.

Develop high-performance Spark applications using DataFrames, Spark SQL, and Structured Streaming.

Create reusable PySpark frameworks and data processing components.

Implement batch and real-time data processing solutions.

Design and manage Data Lake and Lakehouse architectures.

Integrate data from multiple sources including databases, APIs, cloud storage, and streaming platforms.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Implement data quality checks, logging, monitoring, and error handling mechanisms.

Collaborate with Data Scientists, BI teams, and business stakeholders to support analytics requirements.

Support CI/CD processes and deployment automation.

Ensure adherence to data governance, security, and compliance requirements.

Troubleshoot and resolve production issues related to data pipelines and Spark workloads.