Big Data Sr Developer

Roles & Responsibilities

Big Data Development

• Work extensively with the Hadoop ecosystem including HDFS, MapReduce, HBase, Hive, Spark, Kafka, and Presto.
• Design and build scalable data ingestion and processing pipelines.
• Develop ETL workflows using distributed data processing tools.

Streaming & Real-Time Processing

• Build real-time data processing systems using Spark Streaming and Kafka.
• Handle high-volume data ingestion from multiple heterogeneous sources.

Data Engineering & Pipelines

• Develop data ingestion pipelines using tools like Kafka, Flume, Sqoop, and Spark Streaming.
• Integrate data from various source systems into Hadoop-based platforms.

Development & Coding

• Write clean, efficient, and scalable code using Scala (mandatory), Python, or Java.
• Develop automation scripts for data processing workflows.

Advanced Big Data Tools

• Work with tools like Druid, Elasticsearch, Logstash as needed.
• Adapt dynamically to different big data frameworks and evolving use cases.

CI/CD & Deployment

• Support CI/CD pipelines and cloud-based deployments for big data solutions.