Kafka Developer

Kafka Developer Senior Associate

Our Analytics & Insights Managed Services team brings a unique combination of industry expertise, technology, data management and managedservices experience to create sustained outcomes for our clients and improve business performance. We empower companies to transform their approach to analytics and insights while building your skills in exciting new directions. Have a voice at our table to help design, build, and operate the next generation of data and analytics solutions as an Associate.

Job Requirements and Preferences

Basic Qualifications

Minimum Degree Required: Bachelor's Degree in Engineering, Statistics, Mathematics, Computer Science, Data Science, Economics, or a related quantitative field

Minimum Years of Experience: 25 years of professional experience in analytics, data science, or business intelligence roles Preferred Qualifications

Degree Preferred: Master's Degree in Engineering, Statistics, Data Science, Business Analytics, Economics, or related discipline

Preferred Fields of Study: Data Analytics/Science, Statistics, Management Information Systems, Economics, Computer Science Preferred Knowledge & Skills As a Senior Associate in Data Engineering, you'll design and optimize scalable data pipelines, streaming solutions, and data models across enterprise platforms. You'll work with Kafka for real-time streaming, leverage Dataiku and Spark on Cloudera for advanced data workflows, and use Python/PySpark and SQL for data transformations. You'll also mentor junior engineers, enforce best practices, and collaborate with architects and business stakeholders to ensure robust, efficient, and scalable solutions:

Data Streaming & Real-Time Processing (Kafka) Design and implement streaming data ingestion pipelines using Kafka producers, consumers, and topics Manage partitions, consumer groups, offsets, and retention strategies for scalability and fault tolerance Integrate Kafka with downstream platforms such as Spark, Cloudera, or Snowflake Implement monitoring, error handling, and recovery strategies for streaming workloads

Data Engineering with Dataiku & Spark (Cloudera) Build and optimize ETL/ELT workflows in Dataiku and Spark on Cloudera Develop reusable and modular data pipelines for batch and near real-time workloads Optimize Spark jobs using partitioning, broadcast joins, and caching for large-scale datasets Collaborate with platform teams to ensure efficient execution on Cloudera clusters

Python & PySpark Development Write scalable Python/PySpark scripts for data cleansing, transformation, and enrichment Create reusable frameworks and libraries to accelerate development across teams Lead debugging sessions for distributed jobs and perform advanced performance tuning Mentor associates on PySpark best practices and coding standards

Advanced SQL Development Write and optimize complex SQL queries, stored procedures, and functions for analytical and operational workloads Apply indexing, partitioning, and query tuning techniques to improve performance Lead validation and reconciliation of datasets across systems and pipelines Standardize SQL coding practices for junior developers

Data Modeling & Architecture Design and implement conceptual, logical, and physical data models for enterprise solutions Lead dimensional modeling (star/snowflake schema) and normalization for OLTP/OLAP systems Document metadata, entity relationships, and hierarchies for consistent use across teams Partner with architects to align data models with enterprise strategy and governance

Data Quality & Governance Define validation frameworks and reconciliation strategies for streaming and batch pipelines Collaborate with governance teams to ensure data lineage, cataloging, and compliance standards Proactively monitor and resolve data quality issues impacting downstream analytics

Version Control & CI/CD Practices Apply Git b