Job Summary
Senior Data Engineer with 5+ years of experience to design, build, and optimize scalable data pipelines and streaming solutions. The ideal candidate should have strong expertise in Spark (Streaming & Scala), Databricks, Delta Lake, Airflow, Snowflake, and AWS services, with a solid understanding of data engineering best practices and distributed systems.
Key Responsibilities
- Design and develop scalable batch and real-time data pipelines using Spark (Scala) and Spark Streaming
- Build and manage data workflows using Airflow
- Develop and optimize data solutions on Databricks with Delta Lake
- Integrate and manage data across Snowflake and AWS ecosystem
- Work with AWS services such as S3
- Ensure data quality, reliability, and performance of pipelines
- Collaborate with cross-functional teams to understand data requirements and deliver solutions
- Implement best practices for data governance, security, and privacy (CCPA/GDPR)
- Troubleshoot and optimize performance issues in large-scale distributed systems
Required Skills
- 5+ years of experience in Data Engineering / Big Data
- Strong hands-on experience with:
- Apache Spark (Scala) & Spark Streaming
- Databricks & Delta Lake
- Apache Airflow
- Snowflake
- Solid experience with AWS services:
- Strong programming skills in Java/Scala
- Deep understanding of distributed data processing and ETL/ELT design patterns
- Experience in building high-performance, scalable data pipelines
Good to Have
- Experience with Datadog for monitoring and observability
- Working knowledge in Bash and Python
- Familiarity with SBT (Scala Build Tool)
- Experience with GitHub Actions for CI/CD pipelines
- Understanding of data privacy regulations (CCPA, GDPR)
- Experience in real-time streaming architectures
Soft Skills
- Strong problem-solving and analytical skills
- Excellent communication and stakeholder management
- Ability to work in a fast-paced, collaborative environment
Preferred Qualifications
- Experience in cloud-native data platforms
- Prior experience in handling large-scale data platforms
Scala
