- Role - Kafka Admin
- Years of Experience 8+ years of experience
- Location Hyderabad/Bangalore/Pune.
Job Description
- Expertise in Kafka as an admin and ability to work independently and configure and design integrations using Kafka.
- Must have : EEH Kafka Confluent cloud Kafka.
- Deep Understanding of Kafka Architecture: Knowledge of brokers, topics, partitions, producers, consumers, Zookeeper (or KRaft), replication, offsets, consumer groups, etc.
- Installation and Configuration: Ability to set up Kafka clusters from scratch, including multi-broker clusters, and configure various parameters for performance, security, and reliability
- Monitoring and Alerting: Proficiency in using tools (e.g., Prometheus, Grafana, JMX exporters, Confluent Control Center) to monitor Kafka cluster health, performance metrics (throughput, latency, message size, disk usage), and set up effective alerts for issues.
- Troubleshooting and Performance Tuning: The ability to diagnose and resolve common Kafka issues (e.g., consumer lag, network problems, disk I/O bottlenecks, out-of-memory errors),
- SSL/TLS), authorization (ACLs), encryption, and data masking where applicable.
- Backup and Disaster Recovery: Designing and implementing strategies for backing up Kafka data and ensuring business continuity in case of failures. This might involve cross-datacenter replication or specific DR solutions.
- Upgrade Management: Planning and executing upgrades of Kafka brokers and client libraries with minimal downtime.
- Operational Best Practices: Experience with automating Kafka operations, scripting common tasks, and implementing best practices for maintaining a stable and efficient Kafka environment.
- Capacity Planning: Understanding how to project future Kafka usage and plan for scaling the cluster to meet demand.
- Understanding of Kafka Connect: Expertise in using and configuring Kafka Connect for integrating Kafka with various data sources and sinks (databases, data lakes, external APIs, etc.). This includes developing or utilizing custom connectors.
- Schema Registry: Knowledge of Confluent Schema Registry for managing data schemas and ensuring data compatibility. This is crucial for robust data pipelines.
- Kafka Streams/ksqlDB (potentially): While not explicitly stated, someone designing integrations might need familiarity with Kafka Streams or ksqlDB for real-time data processing and transformations before data is sent to its final destination.
- API/Client Library Understanding: While not a developer role, a Kafka Admin designing integrations should understand how producers and consumers interact with Kafka, the various client library configurations, and potential pitfalls.
- Architectural Input: The ability to provide input on how new applications or services should integrate with Kafka, including topic design, partitioning strategies, and message formats.
- Enterprise Event Hub (EEH) Kafka: This strongly suggests the company uses or is planning to use a large-scale, enterprise-grade Kafka deployment, possibly with specific architectural patterns or integrations common in large organizations. This might refer to a specific internal implementation or a common industry term for a centralized event backbone within an enterprise. It implies experience with Kafka in a highly critical, high-volume environment.
- Confluent Cloud Experience: This is a very specific and important requirement. Confluent Cloud is a fully managed Kafka service provided by Confluent. This means the candidate must have hands-on experience with:
Managing Kafka clusters within the Confluent Cloud ecosystem.
Understanding Confluent Cloud specific features: This could include Confluent Control Center (for monitoring, governance, and management), Confluent Schema Registry (as a managed service), Confluent Connect (managed connectors), ksqlDB, and various networking and security configurations within Confluent Cloud.
Billing and resource management within Confluent Cloud.
Troubleshooting and optimizing applications connecting to Confluent Cloud.
Best practices for leveraging a managed Kafka service.
