Key Responsibilities
- Design, develop, and optimize data pipelines for efficient data ingestion, transformation, and delivery from various sources to target systems.
- Implement and manage data delivery solutions using cloud platforms like AWS (S3, Glue, Lambda, Redshift), Snowflake, and Google BigQuery.
- Collaborate with data architects, data scientists, and business analysts to understand data requirements and translate them into technical specifications.
- Develop and maintain DDS documents, outlining data sources, transformations, quality checks, and delivery schedules.
- Ensure data quality, integrity, and security throughout the data lifecycle.
- Monitor data pipelines, troubleshoot issues, and implement solutions to ensure continuous data flow.
- Optimize data storage and query performance on cloud data warehouses.
- Implement automation for data delivery processes and monitoring.
- Stay current with new data technologies and best practices in data engineering and cloud platforms.
Required Skills & Qualifications
- Bachelors or Masters degree in Computer Science, Data Engineering, or a related quantitative field.
- 4+ years of experience in data engineering, with a focus on data delivery and warehousing.
- Proven experience with cloud data platforms, specifically:
- AWS:S3, Glue, Lambda, Redshift, or other relevant data services.
- Snowflake:Strong experience with data warehousing, SQL, and performance optimization.
- Google BigQuery:Experience with data warehousing, SQL, and data manipulation.
- Proficient in SQL for complex data querying, manipulation, and optimization.
- Experience with scripting languages (e.g., Python) for data pipeline automation.
- Solid understanding of data warehousing concepts, ETL/ELT processes, and data modeling.
- Experience with version control systems (e.g., Git).
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams
