Data Engineering Associate Advisor

Position Summary:

Data Engineering Advisor demonstrates expertise in data engineering technologies with the focus on engineering, innovation, strategic influence and product mindset. This individual will act as key contributor of the team to design, build, test and deliver large-scale software applications, systems, platforms, services or technologies in the data engineering space. This individual will have the opportunity to work directly with partner IT and business teams, owning and driving major deliverables across all aspects of software delivery.

The candidate will play a key role in automating the processes on Databricks and AWS. They collaborate with business and technology partners in gathering requirements, develop and implement. The individual must have strong analytical and technical skills coupled with the ability to positively influence on delivery of data engineering products.

The applicant will be working in a team that demands innovation, cloud-first, self-service-first, and automation-first mindset coupled with technical excellence.

The applicant will be working with internal and external stakeholders and customers for building solutions as part of Enterprise Data Engineering and will need to demonstrate very strong technical and communication skills.

Delivery - Intermediate delivery skills including the ability to deliver work at a steady, predictable pace to achieve commitments, decompose work assignments into small batch releases and contribute to tradeoff and negotiation discussions.

Domain Expertise - Demonstrated track record of domain expertise including the ability to understand technical concepts necessary to do the job effectively, demonstrate willingness, cooperation, and concern for business issues and possess in-depth knowledge of immediate systems worked on.

Problem Solving - Proven problem solving skills including debugging skills, allowing you to determine source of issues in unfamiliar code or systems and the ability to recognize and solve repetitive problems rather than working around them, recognize mistakes using them as learning opportunities and break down large problems into smaller, more manageable ones

Job Description & Responsibilities:

• The candidate will be responsible to deliver business needs end to end from requirements to development into production.
• Through hands-on engineering approach in Databricks environment, this individual will deliver data engineering toolchains, platform capabilities and reusable patterns.
• The applicant will be responsible to follow software engineering best practices with an automation first approach and continuous learning and improvement mindset.
• The applicant will ensure adherence to enterprise architecture direction and architectural standards.
• The applicant should be able to collaborate in a high-performing team environment, and an ability to influence and be influenced by others.

Experience Required:

• 8 to 11 years of experience in software engineering, building data engineering pipelines, middleware and API development and automation
• More than 3 years of experience in Databricks within an AWS environment
• Data Engineering experience

Experience Desired:

• Expertise in Agile software development principles and patterns
• Expertise in building streaming, batch and event-driven architectures and data pipelines

Primary Skills:

• Expertise in Big data technologies such as Spark, Hadoop, Databricks, Snowflake, EMR, Glue
• Good understanding of Kafka, Kafka Streams, Spark Structured streaming, configuration-driven data transformation and curation
• Expertise in building cloud-native microservices, containers, Kubernetes and platform-as-a-service technologies such as OpenShift, CloudFoundry
• Experience in multi-cloud software-as-a-service products such as Databricks, Snowflake
• Experience in Infrastructure-as-Code (IaC) tools such as terraform and AWS cloudformation
• Experience in messaging systems such as Apache ActiveMQ, WebSphere MQ, Apache Artemis, Kafka, AWS SNS
• Experience in API and microservices stack such as Spring Boot, Quarkus,
• Expertise in Cloud technologies such as AWS Glue, Lambda, S3, Elastic Search, API Gateway, CloudFront
• Experience with one or more of the following programming and scripting languages - Python, Scala, JVM-based languages, or JavaScript, and ability to pick up new languages
• Experience in building CI/CD pipelines using Jenkins, Github Actions
• Strong expertise with source code management and its best practices
• Proficient in self-testing of applications, unit testing and use of mock frameworks, test-driven development (TDD)
• Knowledge on Behavioral Driven Development (BDD) approach

Additional Skills:

• Cloud-based security principles and protocols like OAuth2, JWT, data encryption, hashing data, secret management, etc
• Ability to perform detailed analysis of business problems and technical environments
• Strong oral and written communication skills
• Ability to think strategically, implement iteratively and estimate financial impact of design/architecture alternatives