Data Pipeline Engineer

Responsibilities

Data Pipeline Engineers are expected to be involved from inception of projects, understand requirements, architect, develop, deploy, and maintain data pipelines (ETL / ELT). Typically, they work in a multi-disciplinary squad (we follow Agile!) which involves partnering with program and product managers to expand product offering based on business demands. Design is an iterative process, whether for UX, services or infrastructure. Our goal is to drive up user engagement and adoption of the platform while constantly working towards modernizing and improving platform performance and scalability.

Deployment and maintenance require close interaction with various teams. This requires maintaining a positive and collaborative working relationship with teams within DOE as well as with wider Aladdin developer community. Production support for applications is usually required for issues that cannot be resolved by operations team. Creative and inventive problem-solving skills for reduced turnaround times are highly valued.

Preparing user documentation to maintain both development and operations continuity is integral to the role.

And Ideal candidate would have

• At least 4+ years experience as a data engineer
• Experience in SQL, Sybase, Linux is a must
• Experience coding in two of these languages for server side/data processing is required Java, Python, C++
• 2+ years experience using modern data stack (spark, snowflake, Big Query etc.) on cloud platforms (Azure, GCP, AWS)
• Experience building ETL/ELT pipelines for complex data engineering projects (using Airflow, dbt, Great Expectations would be a plus)
• Experience with Database Modeling, Normalization techniques
• Experience with object-oriented design patterns
• Experience with dev ops tools like Git, Maven, Jenkins, Gitlab CI, Azure DevOps
• Experience with Agile development concepts and related tools
• Ability to trouble shoot and fix performance issues across the codebase and database queries
• Excellent written and verbal communication skills
• Ability to operate in a fast-paced environment
• Strong interpersonal skills with a can-do attitude under challenging circumstances
• BA/BS or equivalent practical experience

Skills that would be a plus

• Perl, ETL tools (Informatica, Talend, dbt etc.)
• Experience with Snowflake or other Cloud Data warehousing products
• Exposure with Workflow management tools such as Airflow
• Exposure to messaging platforms such as Kafka
• Exposure to NoSQL platforms such as Cassandra, MongoDB
• Building and Delivering REST APIs