Pyspark Developer

We are seeking a skilled PySpark Developer with 4-8 years of experience in building scalable data processing and analytics solutions. The ideal candidate will have strong expertise in PySpark, Python, SQL, and Big Data technologies, with hands-on experience in designing, developing, and optimizing ETL/Data Engineering pipelines.

Key Responsibilities

· Design, develop, and maintain scalable data processing solutions using PySpark.

· Build and optimize ETL pipelines for large-scale structured and unstructured datasets.

· Develop data transformation and ingestion workflows using Python and Spark.

· Write efficient and complex SQL queries for data extraction, validation, and reporting.

· Collaborate with business analysts, data architects, and stakeholders to understand data requirements.

· Perform data quality checks, validation, and troubleshooting of data issues.

· Optimize Spark jobs for performance, scalability, and resource utilization.

· Work with cloud and distributed computing environments to process large volumes of data.

· Participate in code reviews and ensure adherence to coding standards and best practices.

· Support production deployments and resolve application issues.

Required Skills

· 4-8 years of experience in Data Engineering or Big Data Development.

· Strong hands-on experience in PySpark and Python.

· Expertise in SQL and relational databases.

· Experience with Apache Spark ecosystem.

· Strong understanding of ETL concepts and data warehousing principles.

· Experience working with large datasets in distributed environments.

· Knowledge of version control tools such as Git.

· Familiarity with Linux/Unix environments.

Preferred Skills

· Experience with cloud platforms such as AWS, Azure, or GCP.

· Knowledge of Hadoop ecosystem components (Hive, HDFS, Yarn).

· Experience with orchestration tools such as Airflow.

· Exposure to data lake and lakehouse architectures.

· Understanding of CI/CD processes and DevOps practices.

Educational Qualification

· Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Nice to Have

· Experience in Banking, Financial Services, Insurance (BFSI), Retail, or Telecom domains.

· Knowledge of Delta Lake, Databricks, or Snowflake.

· Exposure to real-time data processing using Kafka.

• Keywords: PySpark, Python, Spark SQL, ETL, Data Engineering, Hadoop, Hive, Databricks, Azure Data Factory, Airflow, Kafka, SQL, Cloud, Big Data.