Jr . Data Engineer (SQL,PySpark. AWS Glue)

Company Description

AutoFlow Technologies (AutoFlowTech) is a leading provider of process automation solutions, catering to industries such as manufacturing, aviation, automotive, and industrial processes. Our team of experts designs open, scalable, and modular solutions to help organizations improve efficiency and drive innovation. Serving a diverse customer base, from SMEs to Fortune 500 companies, AutoFlowTech focuses on fostering growth and long-term success. Based on a foundation of problem-solving expertise, we specialize in addressing complex challenges for our clients.

Experience:

•2+ years full-time experience in Data engineering and AWS Cloud Technologies

•Mandatory experience in Mid to Expert Level programming capabilities in a large-scale enterprise

•Experience with Distributed Versioning Control environments such as GIT

•Familiarity with development tools - experience on either IntelliJ / Eclipse / VSCode IDE, Build Tool Maven

• Experience on AWS cloud including DMS, Glue , Pyspark

Responsibilities

• Responsible to Ingest data from files, streams and databases. Process the data with Hadoop, Scala, SQL Database, Spark, ML, IoT

• Develop programs in Python & Pyspark and AQL as part of data cleaning and processing

• Responsible to design and develop distributed, high volume, high velocity multi-threaded Ingestion pipeline

• • Provide high operational excellence guaranteeing high availability and platform stability

• Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Pyspark, Glue, and AWS Cloud computing etc.

• Anticipate and resolve complex problems that impact data design

• Assist in critical technical and design decisions during the development process

Key Performance indicators

•Ensure Data Pipeline is robust and is monitored with respect to schemas, distributions and data completeness

•Evaluate the time required for data to be available for analysis to baseline by monitoring the data velocity against baseline

•Maintain and Monitor the Query to report conversion rate

•Maintain and monitor the optimal time needed for data to be reported in form of KPIs

•Identify issues in codes or bugs that have entered the production stage

• Drive performance optimization by developing optimized codes (Pipelines) for efficient resource utilization

Qualifications:

•Bachelor's degree in Computer Science, Software Engineering or a related technical discipline

•Knowledge of any of the programming languages: Python, Scala or Java

•Primary Skill: AWS , AWS Glue , AWS DMS , Cloudwatch , Pyspark , SQL

•Domain Knowledge: Life Insurance domain is preferred . Familiarity with Finance data and subject is highly preferred