Senior Data Engineer - Python/Spark

4-6 Years

Save

Early Applicant

Job Description

Description

Duties and Responsibilities :

Design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation
Build and support data platform on the cloud
Define and implement automation of jobs and testing
Optimize the data pipeline to support workloads and use cases.
Support mission critical applications and near real time data needs from the data platform.
Build and support reusable framework to ingest, integration and provision data.
Capture and publish metadata and new data to subscribed users
Work collaboratively with product managers, data scientists as well as business partners and actively participate in solution thinking sessions
Participate in design and code reviews
Leads data disaster recovery planning and execution with the disaster recovery team and internal team members.
Mentors more junior members of the team
Determines risk of various configurations and options for data administration and is able to communicate issues to managers and team.
Leads and develops internal standards for data administration and data system documentation.
Develops physical data design.
Defines, documents and executes, referential integrity requirements, constraints and techniques.
Identifies index requirements and determines storage requirements

Requirements

Basic Qualifications :

Bachelors degree in computer science, Information Systems, Engineering, or a related field
4+ years of hands-on experience in data engineering and building data pipelines
4+ years of experience in writing complex SQL and Python
Solid hands-on experience with data warehousing concepts and implementations
4+ years of experience with Snowflake or another modern cloud data warehouse
4+ years of hands-on Python development
Understanding of Data modeling and Data warehousing concepts
2+ years experience with cloud computing (Azure)
Familiarity with at least one cloud platform (AWS, Azure, or GCP)
Strong analytical, problem-solving, and communication skills
Retail/e-comm experience

Preferred Qualifications