Sr Data Engineer (Python, PySpark) with AWS

Description

We are seeking a Sr Data Engineer with expertise in Python and PySpark to join our dynamic team in India. The ideal candidate will have extensive experience designing and implementing data solutions on AWS, playing a crucial role in managing our data infrastructure.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Implement ETL processes to extract, transform, and load data from various sources into data lakes and warehouses on AWS.
• Collaborate with data scientists and analysts to understand data requirements and deliver data solutions that meet their needs.
• Optimize existing data processing workflows for performance and cost efficiency.
• Ensure data quality, integrity, and security throughout the data lifecycle.
• Monitor and troubleshoot data pipeline performance issues, implementing solutions as necessary.
• Stay updated with the latest technologies and best practices in data engineering and AWS services.

Skills and Qualifications

• 6-15 years of experience in data engineering or related fields.
• Strong proficiency in Python and PySpark for data processing and analysis.
• Experience with AWS services such as S3, Redshift, Glue, and Lambda.
• Solid understanding of data warehousing concepts and architecture.
• Experience with SQL and NoSQL databases.
• Familiarity with data modeling and schema design.
• Knowledge of big data technologies like Hadoop or Spark is a plus.
• Ability to work collaboratively in a team environment and communicate effectively with stakeholders.
• Strong problem-solving skills and attention to detail.