Aws Data Engineer

JOB DESCRIPTION:

Must have:

• Strong hands-on experience in Python programming and PySpark
• Experience using AWS services (RedShift, Glue, EMR, S3 & Lambda)
• Experience working with Apache Spark and Hadoop ecosystem.
• Experience in writing and optimizing SQL for data manipulations.
• Good Exposure to scheduling tools. Airflow is preferable.
• Must Have Data Warehouse Experience with AWS Redshift or Hive
• Experience in implementing security measures for data protection.
• Expertise to build/test complex data pipelines for ETL processes (batch and near real time)
• Readable documentation of all the components being developed.
• Knowledge of Database technologies for OLTP and OLAP workloads

Good to have:

• Good understanding of Data warehouse and Data Lakes
• Familiarize ETL tools like Netezza or Informatica.
• Experience working with NoSQL databases like DynamoDB or MongoDB.
• Good to have exposure to AWS services (Step Functions, Athena)
• Data Modelling exposure
• Familiarity with Investment Banking domain

Responsibilities:

• Create and maintain optimal data pipeline architecture for efficient and reliable data processing.
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Database management, ETL process, Data modelling, ensuring data quality and integrity.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, ztc.
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.