PySpark Developer

We are seeking a skilled PySpark Developer with strong working experience in Python programming and a focus on PySpark. You must have hands-on experience with AWS Glue and PySpark, along with strong knowledge of popular libraries such as pandas and numpy. This role requires experience in parallel batch processing, AWS Batch, and Step Functions, as well as knowledge of other AWS services like ECS, ECR, Docker, CloudWatch, and Athena.

Roles & Responsibilities:

• Utilize strong working experience in Python programming to develop data processing solutions.
• Apply expertise with the PySpark framework for big data processing.
• Work extensively with AWS Glue and PySpark for building and managing data pipelines.
• Use popular Python libraries such as pandas, numpy, and joblib to enhance data manipulation and processing.
• Conduct parallel batch processing with Python to handle large-scale data.
• Work with AWS Batch and Step Functions for orchestrating and managing data workflows.
• Utilize knowledge of AWS ECS, ECR, and Docker for containerization and deployment.
• Leverage knowledge of CloudWatch, Athena, and other AWS services for monitoring, analytics, and data management.
• Collaborate with data scientists and other engineers to understand requirements and deliver efficient solutions.
• Troubleshoot and optimize data processing jobs for performance and cost efficiency.

Skills Required:

• Strong working experience in Python programming.
• Expertise with PySpark framework.
• Must have experience on AWS Glue and PySpark.
• Strong experience with using pandas, numpy, joblib, and other popular libraries.
• Good working experience with parallel batch processing with Python.
• Good working experience on AWS Batch and Step Functions.
• Good experience on AWS ECS, ECR, Docker.
• Good knowledge on CloudWatch, Athena, and other AWS services.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.

QUALIFICATION:

• Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent practical experience.