Senior PySpark Developer

Job Description:

• We are looking for an experienced Senior PySpark Developer with 8 years of experience in Data Engineering and Big Data technologies
• The ideal candidate should possess strong expertise in PySpark Python Spark SQL ETL development Data Warehousing and Cloud platforms AWS Azure GCP
• The candidate will be responsible for designing developing and optimizing large scale data processing solutions

Key Responsibilities:

• Key Responsibilities
• Design and develop scalable data pipelines using PySpark and Apache Spark
• Build and optimize ETL ELT frameworks for processing large volumes of structured and unstructured data
• Develop high performance Spark applications using Spark SQL DataFrames and RDDs
• Work closely with data architects and business teams to define data requirements
• Implement data quality validation and governance processes
• Optimize Spark jobs for performance memory management and scalability
• Integrate data from various sources including databases APIs cloud storage and streaming platforms
• Support production deployments and resolve performance bottlenecks
• Mentor junior developers and participate in code reviews

Technical Requirements:

• Mandatory Skills
• 8 years of IT experience
• Strong hands on experience in PySpark and Python
• Expertise in Apache Spark Spark SQL DataFrames and ETL Development
• Strong SQL and database concepts
• Experience with Hadoop Ecosystem Hive HDFS YARN
• Experience in Data Warehousing and Data Modeling
• Hands on experience with Unix Linux

Preferred Skills: