AWS, Pyspark
AWS, Pyspark
Cognizant ConsultingThis job is no longer accepting applications
Job Description
Skill: Pyspark, AWS
Exp: 6 to 12 years
Location: Pune
Job Summary
We are seeking an experienced AWS PySpark Developer to design, develop, and optimize large-scale data processing pipelines in the cloud. The ideal candidate will have strong expertise in PySpark, AWS services, and ETL workflows, with a focus on building scalable, high-performance data solutions for analytics and reporting.
Key Responsibilities
Exp: 6 to 12 years
Location: Pune
Job Summary
We are seeking an experienced AWS PySpark Developer to design, develop, and optimize large-scale data processing pipelines in the cloud. The ideal candidate will have strong expertise in PySpark, AWS services, and ETL workflows, with a focus on building scalable, high-performance data solutions for analytics and reporting.
Key Responsibilities
- Design, develop, and maintain ETL pipelines using PySpark on AWS.
- Work with large datasets to perform data cleansing, transformation, and aggregation.
- Optimize Spark jobs for performance and cost efficiency.
- Integrate data from multiple sources including AWS S3, RDS, Redshift, DynamoDB, and APIs.
- Implement data quality checks and ensure compliance with governance policies.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.
- Deploy and monitor jobs using AWS Glue, EMR, Step Functions, or Lambda.
- Troubleshoot and resolve issues in production data pipelines.
- Maintain documentation for data flows, transformations, and architecture.
- 3–7 years of experience in data engineering or big data development.
- Strong proficiency in PySpark and Spark SQL.
- Hands-on experience with AWS services such as:
- S3, Glue, EMR, Lambda, Athena, Redshift, IAM
- Solid understanding of ETL concepts and data warehousing.
- Proficiency in Python for scripting and automation.
- Experience with version control (Git) and CI/CD pipelines.
- Strong problem-solving skills and ability to work in an agile environment.
- Experience with AWS Glue Studio and Glue Catalog.
- Knowledge of Airflow or other orchestration tools.
- Familiarity with streaming data using Kafka or Kinesis.
- Exposure to Terraform or CloudFormation for infrastructure as code.
- Understanding of data security and encryption in AWS.
More Info
About Company
Skills:
REST API., Core Java, Spring Boot, Microservices, AWS, Azure, Angular, React, Java developer
Skills:
Pyspark, Sql
Skills:
Internal Developer Platforms (IDP), Terraform, Devops Automation, AWS CloudFormation, Infrastructure as Code, Enterprise Cloud Platforms, AI-enabled platform automation, GitHub Enterprise, GitHub Actions, Agentic workflows and automation, AWS Organizations, AWS Control Tower, Platform Engineering
Skills:
CSS3 AWS: Lambda, Html5, S3, Api Gateway, Jenkins, Golang Angular v9+, JavaScript/TypeScript, RDS CI/CD using GitLab CI, or GitHub Actions RESTful APIs
Skills:
snowflake , Informatica, Data Warehousing, Redshift, Databricks, Talend, Python, Etl, Aws S3, Airflow, AWS EMR, Luigi, Business intelligence, Azkaban, Athena