Senior Data Engineer with AWS
Key Responsibilities:
- Design and implement robust ETL pipelines using AWS Glue, Lambda, and S3
- Monitor and optimize the performance of data workflows and batch processing jobs
- Troubleshoot and resolve issues related to data pipeline failures, inconsistencies, and performance bottlenecks
- Collaborate with cross-functional teams to define data requirements and ensure data quality and accuracy
- Develop and maintain automated solutions for data transformation, migration, and integration tasks
- Implement best practices for data security, data governance, and compliance within AWS environments
- Continuously improve and optimize AWS Glue jobs, Lambda functions, and S3 storage management
- Maintain comprehensive documentation for data pipeline architecture, job schedules, and issue resolution processes. Required Skills and Experience:
- Strong experience with Data Engineering practices
- Experience in AWS services, particularly AWS Glue, Lambda, S3, and other AWS data tools
- Proficiency in SQL, python , Pyspark, numpy etc and experience in working with large-scale data sets
- Experience in designing and implementing ETL pipelines in cloud environments
- Expertise in troubleshooting and optimizing data processing workflows
- Familiarity with data warehousing concepts and cloud-native data architecture
- Knowledge of automation and orchestration tools in a cloud-based environment
- Strong problem-solving skills and the ability to debug and improve the performance of data jobs
- Excellent communication skills and the ability to work collaboratively with cross-functional teams
- Good to have knowledge of DBT & SnowflakePreferred Qualifications:
- Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field
- Experience with other AWS data services like Redshift, Athena, or Kinesis
- Familiarity with Python or other scripting languages for data engineering tasks
- Experience with containerization and orchestration tools like Docker or Kubernetes
Location: Candidates should be based in
Gurgaon/Hyderabad
