Data Engineer (AWS+Pyspark)
- Posted 13 hours ago
- Be among the first 10 applicants
Job Description
Job Description
Data Engineer (AWS + pySpark)
Data Engineer (AWS + pySpark)
- Having 4-5 yrs years of relevant experience, which includes hands on experience in Big Data technologies.
- Mandatory - Hands on experience in Python and PySpark.
- Build pySpark applications using Spark Dataframes in Python.
- Worked on optimizing spark jobs that processes huge volumes of data.
- Hands on experience in version control tools like Git.
- Worked on Amazon's Analytics services like Amazon EMR, Amazon Athena, AWS Glue.
- Worked on Amazon's Compute services like Amazon Lambda, Amazon EC2 and Amazon's Storage service like S3 and few other services like SNS.
- Good to have knowledge of datawarehousing concepts – dimensions, facts, schemas- snowflake, star etc.
- Have worked with columnar storage formats - Parquet etc. Well versed with compression techniques – Snappy, Gzip.
- Good to have knowledge of AWS databases (atleast one) Aurora, RDS, Redshift, ElastiCache, DynamoDB.am
More Info
Key Skills
Amazon Athena
Amazon Lambda
Spark Dataframes
About Company
Skills:
amazon emr , S3, RDS, Amazon Ec2, Pyspark, AWS Glue, Dynamodb, Redshift, Git, Sns, Python, AWS, Parquet, Amazon Athena, ElastiCache, Snappy, Spark Dataframes, Aurora, Gzip, Amazon Lambda
Skills:
Git, Sql, AWS, Gitlab, Bitbucket, Python, Jenkins, Pyspark, CodePipeline, Palantir, CodeBuild
Skills:
statistical data analysis , Databricks, Spring Boot, Emr, Java, AWS Glue, Data Warehousing, Hadoop, Pyspark, Kafka, Python, Kubernetes, AWS EKS, Flask, Spark, ETL data pipelines, Flink, Data lakes, Data frameworks, Data engineering concepts, Real-time data processing, NoSQL databases