Job Title: Kafka Developer
Job Type: Fulltime
Location: Pune
Work regime: Hybrid
Keywords: Kafka, Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.
Position Overview: We are looking for a Kafka Developer to design and implement real-time data ingestion pipelines using Apache Kafka. The role involves integrating with upstream flow record sources, transforming and validating data, and streaming it into a centralized data lake for analytics and operational intelligence
What you will Have:-
Responsibilities:
Key Responsibilities:
- Develop Kafka producers to ingest flow records from upstream systems such as flow record exporters (e.g., IPFIX-compatible probes).
- Build Kafka consumers to stream data into Spark Structured Streaming jobs and downstream data lakes.
- Define and manage Kafka topic schemas using Avro and Schema Registry for schema evolution.
- Implement message serialization, transformation, enrichment, and validation logic within the streaming pipeline.
- Ensure exactly once processing, checkpointing, and fault tolerance in streaming jobs.
- Integrate with downstream systems such as HDFS or Parquet-based data lakes, ensuring compatibility with ingestion standards.
- Collaborate with Kafka administrators to align topic configurations, retention policies, and security protocols.
- Participate in code reviews, unit testing, and performance tuning to ensure high-quality deliverables.
- Document pipeline architecture, data flow logic, and operational procedures for handover and support.
Required Skills & Qualifications:
- Proven experience in developing Kafka producers and consumers for real-time data ingestion pipelines.
- Strong hands-on expertise in Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.
- Proficiency in Apache Spark (Structured Streaming) for real-time data transformation and enrichment.
- Solid understanding of IPFIX, NetFlow, and network flow data formats; experience integrating with nProbe Cento is a plus.
- Experience with Avro, JSON, or Protobuf for message serialization and schema evolution.
- Familiarity with Cloudera Data Platform components such as HDFS, Hive, YARN, and Knox.
- Experience integrating Kafka pipelines with data lakes or warehouses using Parquet or Delta formats.
- Strong programming skills in Scala, Java, or Python for stream processing and data engineering tasks.
- Knowledge of Kafka security protocols including TLS/SSL, Kerberos, and access control via Apache Ranger.
- Experience with monitoring and logging tools such as Prometheus, Grafana, and Splunk.
- Understanding of CI/CD pipelines, Git-based workflows, and containerization (Docker/Kubernetes)
A little about us:
Innova Solutions is a diverse and award-winning global technology services partner. We provide our clients with strategic technology, talent, and business transformation solutions, enabling them to be leaders in their field.
- Founded in 1998, headquartered in Atlanta (Duluth), Georgia.
- Employs over 50,000 professionals worldwide, with annual revenue approaching $3.0B.
- Delivers strategic technology and business transformation solutions globally.
- Operates through global delivery centers across North America, Asia, and Europe.
- Provides services for data center migration and workload development for cloud service providers.
Awardee of prestigious recognitions including:
- Women's Choice Awards - Best Companies to Work for Women & Millennials, 2024
- Forbes, America's Best Temporary Staffing and Best Professional Recruiting Firms, 2023
- American Best in Business, Globee Awards, Healthcare Vulnerability Technology Solutions, 2023
- Global Health & Pharma, Best Full Service Workforce Lifecycle Management Enterprise, 2023
- Received 3 SBU Leadership in Business Awards
- Stevie International Business Awards, Denials Remediation Healthcare Technology Solutions, 2023
