Open Location - Indore, Noida, Gurgaon, Bangalore, Hyderabad, Pune
Job Description
- 3-8 years experience working on Data engineering & ETL/ELT processes, data warehousing, and data lake implementation with AWS services
- Hands on experience in designing and implementing solutions like creating/deploying jobs, Orchestrating the job/pipeline and infrastructure configurations
- Expertise in designing and implementing pySpark and Spark SQL based solutions
- Design and implement data warehouses using Amazon Redshift, ensuring optimal performance and cost efficiency.
- Good understanding of security, compliance, and governance standards.
Roles & Responsibilities
- Design and implement robust and scalable data pipelines using AWS/Azure services
- Drive architectural decisions for data solutions on AWS, ensuring scalability, security, and cost-effectiveness.
- Hands-on experience of Develop and deploy ETL/ELT processes using Glue/Azure data factory, Lambda/Azure functions, Step function/Azure logic apps/MWAA, S3 and Lake formation from various data sources.
- Strong Proficiency in pySpark, SQL, Python.
- Proficiency in SQL for data querying and manipulation.
- Experience with data modelling, ETL processes, and data warehousing concepts.
- Create and maintain documentation for data pipelines, processes, and following best practices.
- Knowledge of various Spark Optimization technique, Monitoring and Automation would be a plus.
- Participate in code reviews and ensure adherence to coding standards and best practices.
- Understanding of data governance, compliance, and security best practices.
- Strong problem-solving and troubleshooting skills.
- Excellent communication and collaboration skills with understanding on stakeholder mapping
Good to Have:
- Understanding of databricks is good to have.
- GenAI, Working with LLMs are good to have
Mandatory Skills - AWS OR Azure Cloud, Python Programming, SQL, Spark SQL, Hive, Spark optimization techniques and Pyspark.
Share resume at [Confidential Information] with details (CTC, Expected CTC, Notice Period)
