Hadoop Data Engineer

• Design, develop, and maintain big data pipelines using Hadoop ecosystem technologies such as HDFS, Hive, and Spark.

• Build scalable data processing solutions using Spark and Python for large-scale data handling.

• Work on Hadoop to AWS migration projects ensuring seamless data transition and performance optimization.

• Develop and maintain Unix shell scripts for automation of data workflows and system processes.

• Collaborate with cross-functional teams to design, develop, and implement data engineering solutions.

• Participate in Agile development cycles including design, development, testing, and deployment activities.

• Ensure data quality, integrity, and performance across distributed data systems.

• Optimize existing big data applications for better performance and scalability.

• Support application design, software development, and testing processes.

• Troubleshoot and resolve issues in big data pipelines and processing systems.