- Posted 2 months ago
- Be among the first 30 applicants
Job Description
Senior Data Engineer (AWS & Snowflake Specialist)
Experience Level: 5–6 Years
Role Overview
This role is for a data powerhouse who can bridge the gap between high-performance big
data processing on AWS and elite-level data warehousing in Snowflake. You will be
responsible for the entire lifecycle of data, ensuring that every pipeline includes robust
validation and observability.
Key Responsibilities
Big Data Processing: Leverage AWS EMR (Spark/Hive) for heavy-lift data
processing and Athena for serverless ad-hoc querying of S3 data lakes.
Data Validation: Implement Great Expectations (GX) or equivalent Python-based
validation libraries to prevent data silent failure within EMR and Snowflake ingestion
layers.
Snowflake Engineering: Architect end-to-end Snowflake environments, utilizing
SnowSQL for scripting and SnowPipe for continuous, automated data ingestion.
System Integration: Build and maintain Snowflake Integrations (Storage
Integrations with S3, API Integrations, and Notification Services).
Advanced ETL: Develop robust pipelines that transition data from raw AWS storage
into optimized Snowflake schemas.
Technical Requirements
Core Languages (Strong Proficiency Required):
SQL: Expert mastery of Snowflake-specific SQL extensions, CTEs, and
performance tuning (Micro-partitioning).
Python: Advanced scripting for automation, Boto3 for AWS resource
management, and data manipulation.
Data Quality: Proven experience implementing automated data testing and
validation frameworks within production pipelines.
AWS Stack: Extensive experience with EMR, Redshift, Athena, S3, and IAM
security policies.
Snowflake Mastery: * Proven experience with Snowflake Data Engineering
features (Streams, Tasks, and External Tables).
Hands-on usage of SnowPipe for real-time or near-real-time data loading.
More Info
About Company
