Kafka Flink Engineer

Job Title: Confluent Flink Engineer

Experience: 6+ Years

Location: Remote

Shift Timings: 2:00 PM – 11:00 PM IST

Role Overview

We are seeking an experienced Confluent Flink Engineer with strong expertise in real-time data streaming, event-driven architectures, and large-scale distributed systems. The ideal candidate should have hands-on experience with Apache Flink, Confluent Kafka, Kafka Streams, and streaming ETL pipelines, along with a strong understanding of cloud-native data processing solutions.

The role involves designing, developing, and optimizing real-time streaming applications that process high-volume data efficiently while ensuring scalability, fault tolerance, and performance.

Key ResponsibilitiesStream Processing & Data Engineering

• Design, develop, and maintain real-time data processing applications using Apache Flink.
• Build and manage large-scale streaming ETL pipelines.
• Develop event-driven architectures using Confluent Kafka and Kafka Streams.
• Implement real-time data transformation, aggregation, enrichment, and analytics workflows.
  
  

Kafka & Confluent Platform

• Configure and manage Confluent Kafka environments.
• Work with Kafka Connect, Schema Registry, and other Confluent Platform components.
• Design efficient topic structures, partitioning strategies, and consumer group architectures.
• Ensure reliable event processing and message delivery.
  
  

Performance & Scalability

• Optimize Flink jobs for performance, throughput, and resource utilization.
• Implement checkpointing, state management, and fault-tolerant processing strategies.
• Monitor and troubleshoot streaming applications and data pipelines.
• Ensure scalability and high availability of streaming platforms.
  
  

Cloud & DevOps

• Deploy and manage streaming workloads on AWS, Azure, or GCP.
• Participate in CI/CD pipeline implementation and automation.
• Collaborate with DevOps teams to improve deployment, monitoring, and operational practices.
  
  

Collaboration

• Work closely with Data Engineers, Architects, Product Teams, and Business Stakeholders.
• Participate in architecture discussions and technical design reviews.
• Contribute to best practices, documentation, and knowledge sharing.
  
  

Required Skills & Experience

• 6+ years of software engineering or data engineering experience.
• Strong hands-on experience with Apache Flink.
• Extensive experience with Confluent Kafka and Kafka Streams.
• Strong understanding of real-time streaming and event-driven architectures.
• Experience with distributed systems and large-scale data processing.
• Proficiency in Java, Scala, or Python.
• Experience building and managing streaming ETL pipelines.
• Strong knowledge of:
• Kafka Connect
• Schema Registry
• Confluent Platform Components
• Strong understanding of:
• Data Partitioning
• State Management
• Checkpointing
• Fault Tolerance
• Experience with AWS, Azure, or GCP cloud platforms.
• Familiarity with CI/CD pipelines and DevOps practices.
  

Preferred Skills

• Experience with Kubernetes and containerized deployments.
• Knowledge of data lake and modern data platform architectures.
• Exposure to monitoring tools such as Prometheus, Grafana, or Datadog.
• Experience with Spark Streaming or other stream processing technologies.
  
  

Key Competencies

• Real-Time Data Streaming
• Apache Flink Development
• Confluent Kafka & Kafka Streams
• Distributed Systems
• Event-Driven Architecture
• Streaming ETL Pipelines
• Cloud Data Engineering
• Performance Optimization
• CI/CD & DevOps
  
  

Skills: kafka,data,processing,etl,devops,cloud,pipelines,flink