Skill: Python, Pyspark, AWS
Exp: 6 to 12 years
Location: Pune
Job Summary
We are seeking a skilled
Python + PySpark + AWS Developer to design, develop, and maintain scalable data engineering solutions. The role involves building high-performance ETL pipelines, processing large datasets, and integrating with AWS cloud services to support enterprise analytics and reporting needs.
Key Responsibilities
- Design, develop, and optimize ETL pipelines using PySpark and Python.
- Work with large-scale datasets to perform data cleansing, transformation, and aggregation.
- Integrate with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, Kinesis.
- Implement data lake and data warehouse solutions on AWS.
- Ensure data quality, consistency, and security across all processing stages.
- Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions.
- Monitor, troubleshoot, and optimize data workflows for performance and cost efficiency.
- Follow best practices for code versioning, CI/CD, and cloud resource management.
- Write unit tests and maintain documentation for all developed solutions.
Required Skills & Qualifications
- Strong programming skills in Python (data manipulation, APIs, automation).
- Hands-on experience with PySpark for distributed data processing.
- Proficiency in AWS cloud services (S3, Glue, EMR, Lambda, Redshift, Athena, IAM).
- Solid understanding of SQL and relational database concepts.
- Experience with data lake and data warehouse architectures.
- Knowledge of ETL/ELT design patterns and best practices.
- Familiarity with Git and CI/CD tools (e.g., Jenkins, GitHub Actions).
- Strong problem-solving and debugging skills.
- Bachelor's degree in Computer Science, Information Technology, or related field.
Preferred Skills
- Experience with streaming data (Kafka, Kinesis).
- Knowledge of Docker and containerized deployments.
- Familiarity with Terraform or CloudFormation for infrastructure as code.
- Exposure to machine learning pipelines in a big data environment.
- Understanding of data governance and compliance standards.
Work Environment
- Location: Onsite / Remote / Hybrid (as per company policy)
- Team: Works closely with data engineers, analysts, and cloud architects.
- Tools: AWS Cloud, Spark, Python, SQL, Git, CI/CD pipelines.
