Job Description
Senior Data Engineer
JD Design develop and maintain scalable ETL pipelines using AWS Glue PySpark and AWS Lambda
Migrate structured and semistructured data from existing data lakes eg S3 Hadoop to Amazon Redshift
Optimize data ingestion and transformation processes for performance and costefficiency
Collaborate with data architects analysts and business stakeholders to understand data requirements and ensure data quality
Implement data validation cleansing and transformation logic
Monitor and troubleshoot data pipelines and workflows
Ensure data security and compliance with organizational and regulatory standards
Document data flows architecture and processes Responsibility
Design develop and maintain scalable ETL pipelines using AWS Glue PySpark and AWS Lambda
Migrate structured and semistructured data from existing data lakes eg S3 Hadoop to Amazon Redshift
Optimize data ingestion and transformation processes for performance and costefficiency
Primary Skills
AWS Glue jobs
AWS services Lambda S3
AWS database Hadoop and Redshift
Communication
Usage of AI code assist tools
Working knowledge of implementing data migration
