1. Designing and building data pipelines to collect, process, transform, and store data from various sources. This involves understanding the data flow, selecting the appropriate technologies, and creating scalable and efficient data processing workflows.

2. Ensure that the data pipeline architecture is reliable and produces high-quality data. This involves monitoring the data pipeline, identifying and resolving issues, and implementing data validation and testing mechanisms.

3. Work closely with data scientists, data analysts, software engineers, and other stakeholders to understand their data requirements and ensure that the data pipeline architecture meets their needs.

4. Optimizing the data pipeline architecture to ensure efficient and scalable data processing. This involves identifying bottlenecks, optimizing data processing workflows, and selecting appropriate technologies and tools.

5. Ensure security and privacy of data in the pipeline architecture. This involves implementing appropriate security measures, such as data encryption, access controls, and auditing.

6. Documentation of the data pipeline architecture, data processing workflows, and data sources. This involves creating technical documentation, data flow diagrams, and reports that communicate the performance and quality of the data pipeline to stakeholders.

Qualification and Experience:

Qualification:

A bachelors in computer science, data science, software engineering, or a related field

Experience:

5+ years of relevant experience in data engineering, software development, or data architecture, in a cloud-based environment.
Technical skills in data engineering, software development, and data architecture. This includes experience with big data technologies.
Proficiency in one or more programming languages such as Python, Java, or Scala, as well as experience with scripting languages like Bash or PowerShell.
Data management skills, data modelling, data governance, and data security. Analytical skills and experience with data analysis, data profiling, and data quality assessment.
Project management methodologies, such as Agile or Scrum, and be able to manage projects independently.

Data Pipeline Engineer