r data infrastructure.

Responsibilities

Design, develop, and maintain scalable data pipelines using Python and PySpark.
Implement ETL processes to extract, transform, and load data from various sources into data lakes and warehouses on AWS.
Collaborate with data scientists and analysts to understand data requirements and deliver data solutions that meet their needs.
Optimize existing data processing workflows for performance and cost efficiency.
Ensure data quality, integrity, and security throughout the data lifecycle.
Monitor and troubleshoot data pipeline performance issues, implementing solutions as necessary.
Stay updated with the latest technologies and best practices in data engineering and AWS services.

Skills and Qualifications

6-15 years of experience in data engineering or related fields.
Strong proficiency in Python and PySpark for data processing and analysis.
Experience with AWS services such as S3, Redshift, Glue, and Lambda.
Solid understanding of data warehousing concepts and architecture.
Experience with SQL and NoSQL databases.
Familiarity with data modeling and schema design.
Knowledge of big data technologies like Hadoop or Spark is a plus.
Ability to work collaboratively in a team environment and communicate effectively with stakeholders.
Strong problem-solving skills and attention to detail.