Data Engineer
Data Engineer
redseer strategy consultantsThis job is no longer accepting applications
Job Description
Brief Description
At-least 1 year of Python, Spark, SQL, data engineering experience
Primary Skillset: PySpark, Scala/Python/Spark, Azure Synapse, S3, RedShift/Snowflake
Relevant Experience: Legacy ETL job Migration to AWS Glue / Python & Spark combination
Role Scope
Reverse engineer the existing/legacy ETL jobs
Create the workflow diagrams and review the logic diagrams with Tech Leads
Write equivalent logic in Python & Spark
Unit test the Glue jobs and certify the data loads before passing to system testing
Follow the best practices, enable appropriate audit & control mechanism
Analytically skillful, identify the root causes quickly and efficiently debug issues
Take ownership of the deliverables and support the deployments
Requirements
Create data pipelines for data integration into Cloud stacks eg. Azure Synapse
Code data processing jobs in Azure Synapse Analytics, Python, and Spark
Experience in dealing with structured, semi-structured, and unstructured data in batch and real-time environments.
Should be able to process .json, .parquet and .avro files
Preferred Background
Tier1/2 candidates from IIT/NIT/IIITs
However, Relevant Experience, Learning Attitude Takes Precedence
Skills:- Python, PySpark, SQL, pandas, Cloud Computing, Microsoft Windows Azure and Big Data
At-least 1 year of Python, Spark, SQL, data engineering experience
Primary Skillset: PySpark, Scala/Python/Spark, Azure Synapse, S3, RedShift/Snowflake
Relevant Experience: Legacy ETL job Migration to AWS Glue / Python & Spark combination
Role Scope
Reverse engineer the existing/legacy ETL jobs
Create the workflow diagrams and review the logic diagrams with Tech Leads
Write equivalent logic in Python & Spark
Unit test the Glue jobs and certify the data loads before passing to system testing
Follow the best practices, enable appropriate audit & control mechanism
Analytically skillful, identify the root causes quickly and efficiently debug issues
Take ownership of the deliverables and support the deployments
Requirements
Create data pipelines for data integration into Cloud stacks eg. Azure Synapse
Code data processing jobs in Azure Synapse Analytics, Python, and Spark
Experience in dealing with structured, semi-structured, and unstructured data in batch and real-time environments.
Should be able to process .json, .parquet and .avro files
Preferred Background
Tier1/2 candidates from IIT/NIT/IIITs
However, Relevant Experience, Learning Attitude Takes Precedence
Skills:- Python, PySpark, SQL, pandas, Cloud Computing, Microsoft Windows Azure and Big Data
More Info
About Company
Skills:
AWS Athena, Sql, ELT, AWS IAM, Aws Redshift, Aws S3, AWS Glue, Data Warehousing, Apache Spark, Aws Lambda, Pyspark, Etl, Python, Git, Data Modeling, Airflow, CI CD, AWS EMR, AWS Step Functions
Skills:
Power Bi, Power Automate, Power Query, Sql, Microsoft 365, Dax, Data Analytics, Python, Root Cause Analysis, Process Automation, Dashboard Development, Sharepoint, KPI Reporting, ETL Data Pipelines
Skills:
data engineering , snowflake , Cloud Computing, Cloud Integration, Technical Solution Delivery, Data and AI use case implementation
Skills:
Power Bi, Pyspark, Agile Scrum, Tableau, Databases, Git, Data Warehousing, Informatica Powercenter, Python, Airflow, Data Lake concepts, Advanced SQL and performance tuning, IICS, dbt, CI/CD