Role Summary
The Senior Data Engineer will design, build, and optimize large-scale data pipelines and data platforms. This role is hands-on and plays a key part in translating architectural designs into production-ready solutions.
Key Responsibilities
- Build and maintain scalable batch and streaming data pipelines
- Optimize Spark jobs for performance and reliability
- Develop and manage data models and transformations
- Ensure data quality, reliability, and observability
- Collaborate with architects to implement target architectures
- Ensure smooth deployment to higher environments
- Review code, enforce engineering standards, and mentor junior engineers
- Support production deployments and incident resolution
Core Technical Skills
- Strong experience in Spark, PySpark and Python
- Strong experience in designing and maintaining ETL pipelines.
- Experience with scheduling ETL pipelines, defining dependencies
- Experience in scheduling ETL pipelines and knowledge of Apache Airflow.
- Strong RDBMS & SQL skills
- Experience in dimensional data modelling/3NF data modelling for Data warehouses.
- Experience in implementation of Data Lakehosue/Data Lake/Data Warehouse
- Strong experience in one of the major cloud platforms. Preferably AWS, Azure.
Experience
- 410 years of experience in data engineering roles
- Exposure to product engineering backgrounds is a strong plus, with emphasis on building scalable, data-driven platforms.
