Python, Pyspark, AWS

Skill: Python, Pyspark, AWS

Exp: 6 to 12 years

Location: Pune

Job Summary

We are seeking a skilled Python + PySpark + AWS Developer to design, develop, and maintain scalable data engineering solutions. The role involves building high-performance ETL pipelines, processing large datasets, and integrating with AWS cloud services to support enterprise analytics and reporting needs.

Key Responsibilities

• Design, develop, and optimize ETL pipelines using PySpark and Python.
• Work with large-scale datasets to perform data cleansing, transformation, and aggregation.
• Integrate with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, Kinesis.
• Implement data lake and data warehouse solutions on AWS.
• Ensure data quality, consistency, and security across all processing stages.
• Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions.
• Monitor, troubleshoot, and optimize data workflows for performance and cost efficiency.
• Follow best practices for code versioning, CI/CD, and cloud resource management.
• Write unit tests and maintain documentation for all developed solutions.
  
  

Required Skills & Qualifications

• Strong programming skills in Python (data manipulation, APIs, automation).
• Hands-on experience with PySpark for distributed data processing.
• Proficiency in AWS cloud services (S3, Glue, EMR, Lambda, Redshift, Athena, IAM).
• Solid understanding of SQL and relational database concepts.
• Experience with data lake and data warehouse architectures.
• Knowledge of ETL/ELT design patterns and best practices.
• Familiarity with Git and CI/CD tools (e.g., Jenkins, GitHub Actions).
• Strong problem-solving and debugging skills.
• Bachelor's degree in Computer Science, Information Technology, or related field.
  
  

Preferred Skills

• Experience with streaming data (Kafka, Kinesis).
• Knowledge of Docker and containerized deployments.
• Familiarity with Terraform or CloudFormation for infrastructure as code.
• Exposure to machine learning pipelines in a big data environment.
• Understanding of data governance and compliance standards.
  
  

Work Environment

• Location: Onsite / Remote / Hybrid (as per company policy)
• Team: Works closely with data engineers, analysts, and cloud architects.
• Tools: AWS Cloud, Spark, Python, SQL, Git, CI/CD pipelines.