Data Engineer - Treasure Data

3-5 Years

Save

Early Applicant

Job Description

Key Result Areas And Activities

Data Pipeline Development and Maintenance:
Design, develop, and maintain scalable data pipelines to ensure efficient data flow and processing.
Implement ETL (Extract, Transform, Load) processes to integrate data from various sources.
Data Quality and Integrity:
Ensure the accuracy, completeness, and reliability of data by implementing robust data validation and cleansing processes.
Monitor data quality metrics and address any discrepancies or issues promptly.
Performance Optimization:
Optimize data storage and retrieval processes to enhance system performance and reduce latency.
Implement indexing, partitioning, and other techniques to improve query performance.
Collaboration and Communication:
Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
Communicate technical concepts and findings effectively to non-technical team members.
Security and Compliance:
Implement data security measures to protect sensitive information and ensure compliance with relevant regulations and standards.
Conduct regular audits and assessments to identify and mitigate potential security risks.

Essential Skills

Understanding of enterprise data warehousing principles of ETL/ELT and the tools/technologies used to implement them
Experience with Treasure Data, including designing and maintaining complex workflows.
Strong knowledge of Dig Dag files
Strong knowledge of Presto SQL and Optimisation techniques
Experience with Python and relational databases
Familiarity with cloud platforms (e.g., AWS) and their data services.

Desirable Skills