Senior Data Engineer

This role is based in Chennai, India.

The Senior Data Engineer role will have line management and reporting into the VP Technology Chapter, but will be assigned to a Platform squad for task management and prioritisation. The engineer will implement the design and build of the platform along with all associated components from inception to maturity of the running environment over time. The role will be responsible for driving change and delivering new technology into the Bank by onboarding customers and applications to the Platform via self-service and automation.

The engineer will work closely with other Platforms and Product teams in educating them on the new Platform, services and features to ensure any transition and consumption is optimal. Teams work using Agile + DevOps + Product Management methodologies, delivering work packages based on the priority in the backlog.

Roles & Responsibilities:

• Hands on Tier – 2 production experience with Cloudera:

Installing and maintenance

Enabling security requirements for CDP

Optimizing cluster performance

Working with data engineers to optimize Spark job performance

Automation – Ansible, data/analytics jobs CI/CD

Observability

• Installation, configuration and maintenance of the CDP latest version of CDP 7.1.9 or CDP latest version experience with Cloudera data services 1.5.4 and 1.5.5
• Trouble shooting the Spark job , Hive Jobs and Yarn jobs
• CDP Containerization with data services
• experience in writing automation using Shell, Terraform, Groovy and Ansible.
• Automated pipelines/workflows (GitHub Actions, Jenkins, ArgoCD etc.) for data projects. Python based preferred.
• Automating security/regulatory checks
• Automated patching
• Containerization – Podman, Docker, OCP ,
• Container Management – Kubernetes/OpenShift
• Linux OS & Networking basics
• Source control (git)
• Agile engineering and practices
• Trustful and professional relationships with key stakeholders
• Has developed synergies across services, optimizing re-use, promoting simplification and demonstrating speed-to-market benefits
• Commercial and business focused to improve IT services
• Trustful and pragmatic approach to problem solving
• Ability to adopt the Group's values and drive adoption of the values across the team
• Ability to work on projects from inception to delivery
• Decision making to decide on best solution based on cost, time and quality of options
• Cross skills in delivery but also operational support as per the new DevOps model
• Implementing robust solutions to minimize operational effort

Required Qualifications:

• IT or similar bachelor's degree or higher
• Equivalent work experience
• At least 4 years production Hadoop experience
• 8+ years hands on Technology experience
• Experience with large and complex global enterprises defined by high availability, broad observability, and high level of reliability.
• At least 1 project experience in Data Science project/products
• Excellent communication and articulation skills
• Hands on Tier – 2 production experience with Cloudera:
• Installing and maintenance
• Optimizing cluster performance
• Working with data engineers to optimize Spark job performance
• Automation – Ansible, data/analytics jobs CI/CD
• Observability
• Automated pipelines/workflows (GitHub Actions, Jenkins, ArgoCD etc.) for data projects. Python based preferred.
• Some Python programming experience
• Shell, Groovy scripting experience
• Automating security/regulatory checks
• Automated patching
• Containerization – Podman, Docker
• Container Management – Kubernetes/OpenShift
• Linux OS & Networking basics
• Source control (git)

Nice to have:

• Public cloud exposure
• Data Science projects/products
• Collaboration
• Self-starter with strong initiative
• Innovative and experimental mindset
• Calm under pressure
• Pragmatic and focused