Engineering Manager Data Engineering

About The Job :

This is an opportunity in which you will be responsible for overall implementation and delivery of projects and programmes, bringing new capabilities to the market, or implementing large scale change to existing products, covering the full lifecycle. You will oversee/manage the technical

delivery management for a business area. You will supervise and coordinate the development teams (both internal employees and external partners) during their implementation. You will work closely with the businesses to understand their business objectives, product strategies and roadmaps and requirements so they can be continually translated into priorThis is an opportunity in which you will be responsible for overall implementation and delivery of projects and programmes, bringing new capabilities to the market, or implementing large scale change to existing products, covering the full lifecycle. You will oversee/manage the technical

delivery management for a business area. You will supervise and coordinate the development teams (both internal employees and external partners) during their implementation. You will work closely with the businesses to understand their business objectives, product strategies and roadmaps and requirements so they can be continually translated into prioritised technology deliverables.

Web development, with experience in JavaScript frameworks

Cloud Integrations, Security

ECS/EKS, API Gateways, Route53/F5, VPC/ELB, REST / SOAP frameworks

Airflow, Glue, Lambdas, Redshift – with Node/Python.

Liaise with business/data analysts and business stakeholders to review new requirements and enhancements and help identify the best technical solution for the data ingestion/distribution tasks

Design data extraction and transformation logic based on prevailing ETL/ELT technologies, languages and open-source frameworks (e.g., Python, Airflow, AWS Data Wrangler, AWS Sage Maker, AWS EMR, AWS Glue etc.)

Supervise, coach and monitor the development teams and vendors in the implementation of designed data pipelines and transformations

Ensure all the data extraction and transformation jobs are successfully scheduled and executed, to guarantee data timeliness and quality

Design, build and run data quality metrics for early identification of anomalies and erroneous data pipeline executions

Report progress, risks and issues to the line manager and other relevant stakeholders (e.g., business users, senior management team, leadership team etc)

Evaluate new trends and technologies in the market and prepare suggestions and

recommendations to be presented to the line manager and other members of the senior management team

Drive a culture of collaborative reviews of designs, code and test plans.

Ability to arrive at actionable insights leveraging the pro-active or pre-emptive measures and algorithms around the data driven decision.

What we're looking for:

• Excellent understanding of enterprise data warehousing principles and ETL/ELT solutions
• Flexible and able to adjust to changing priorities in a fast-paced environment.
• Experience working on or with API's for data delivery/ingestion, e.g., REST APIs, GraphQL, etc.
• Experience working on or with batch-based data delivery/ingestion paradigms, e.g., csv files, ODBC connections, etc.
• Experience in Code management tools (GitHub) Build and Deployment tool (TravisCI).
• Excellent knowledge and working experience on data management technologies and frameworks, preferably on Python, Airflow, AWS API Gateway, SNS, SQS, Lambda
• Good understanding of Lambda and Kappa Architecture patterns
• Ability to work with large and complex data sets, data lakes and data warehouses
• SQL query skills. Hands-on experience with AWS Redshift, PostgreSQL and other database technologies like MySQL, Microsoft SQL Server, etc. and ability to write complex queries to extract/inject data into those databases
• Proven ability to manage a team of developers and to work in an objective oriented environment
• Experience with Agile methodology including WIP management, backlog grooming/prioritization, reporting and dashboards.

Key Skills:-

• Strong hands-on experience with Python
• Apache Airflow or similar workflow orchestration tools
• AWS Data Services – Glue, Lambda, Redshift, EMR
• ETL/ELT, Data Lakes & Data Warehousing
• REST/SOAP APIs and API integrations
• PostgreSQL, MySQL, SQL Server & strong SQL
• AWS API Gateway, SNS, SQS
• ECS/EKS, VPC, ELB, Route53/F5
• JavaScript frameworks / Web Development
• GitHub, CI/CD and Agile methodologies
• Engineering team leadership and stakeholder management