We are looking for a highly skilled
Senior Data Streaming Engineer with strong expertise in
Apache Flink and
Apache Spark to design, build, and optimize enterprise-grade real-time and batch data processing solutions. The ideal candidate will have extensive experience developing streaming applications, event-driven architectures, and high-performance ETL pipelines while ensuring scalability, reliability, and low-latency processing.
Responsibilities
- Design, develop, and maintain scalable real-time data processing pipelines.
- Build streaming applications using Apache Flink DataStream API, Table API, and Spark Structured Streaming.
- Develop batch and near real-time ETL workflows.
- Implement stateful stream processing, event-time processing, watermarking, windowing, checkpointing, and fault-tolerant streaming solutions.
- Optimize Spark and Flink jobs for performance, scalability, and resource utilization.
- Process large-scale datasets with high throughput and low latency.
- Collaborate with architects, data engineers, and business teams to deliver enterprise data solutions.
- Implement monitoring, logging, alerting, and troubleshooting mechanisms.
- Ensure data quality, governance, and security best practices.
Required Skills- 8–10 years of Data Engineering experience.
- Strong hands-on expertise in Apache Flink:
- DataStream API
- Table API
- SQL API
- Stateful Stream Processing
- Event-Time Processing
- Windowing
- Watermarks
- Checkpointing
- Fault Tolerance
- Strong expertise in Apache Spark:
- Spark Core
- Spark SQL
- Structured Streaming
- Experience developing high-volume ETL pipelines.
- Strong understanding of distributed computing concepts.
- Experience with performance tuning and optimization.
- Good programming skills in Java, Scala, or Python.
Preferred Skills
- Kafka or Azure Event Hubs
- Azure, AWS, or GCP
- Docker & Kubernetes
- CI/CD implementation
- Data Lakes
- Microservices Architecture
- Agile development
Skills: etl,apache spark,flink,pipelines
