Skill Set:
AWS, Snowflake, Kafka, Airflow, GitHub, PySpark
Key Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines
- Ingest data from various sources (APIs, databases, files, etc.)
- Implement both real-time and batch processing solutions based on use case requirements
- Ensure data quality through validation and cleansing processes
- Collaborate with Product Managers and Business Stakeholders to gather and understand data requirements
- Translate business needs into technical specifications
- Ensure data security, access control, and compliance with relevant policies
- Maintain documentation and follow best practices for data engineering
Ideal candidate profile: 4-8 years of hands-on experience with Snowflake, Airflow and Pyspark in any cloud platform (AWS/Azure/GCP). The candidate should be serving notice with LWD in the next 3 weeks.
