About Client:
We are partnering with a renowned clienta dynamic and innovative company dedicated to delivering cutting-edge software solutions. They are seeking a skilled and motivated Hadoop Admin to join their growing team and contribute to the development of high-quality, scalable, and efficient software applications.
Job Description:
As a Hadoop Administrator, you will be responsible for managing, configuring, and maintaining the Hadoop ecosystem, including Hadoop Distributed File System (HDFS), YARN, and other related tools. You will ensure the reliability, performance, and scalability of the platform, providing support for data ingestion, processing, and analysis. Additionally, you will collaborate with data engineers, data scientists, and developers to ensure the platform meets the needs of the organization.
Key Responsibilities:
- Hadoop Cluster Management:
- Install, configure, and maintain Hadoop clusters, including HDFS, YARN, MapReduce, Hive, Pig, and other Hadoop ecosystem components.
- Perform regular cluster monitoring, maintenance, and optimization to ensure high availability and performance.
- Manage the deployment, scaling, and upgrading of Hadoop clusters and its components.
- Ensure proper configuration of cluster resources, including memory, disk, and CPU usage.
- Data Ingestion and Storage:
- Manage data ingestion workflows, ensuring seamless data pipelines and the storage of data in HDFS or other distributed storage systems.
- Optimize HDFS storage, ensuring efficient file distribution, replication, and access control policies.
- Implement and manage data lifecycle policies for data retention and archival.
- Security and Access Control:
- Implement and manage security measures within the Hadoop ecosystem, including Kerberos authentication, LDAP, and access control policies.
- Manage user roles, permissions, and access control to ensure data security and governance.
- Regularly audit the Hadoop environment to ensure compliance with security policies and industry standards.
- Performance Tuning and Optimization:
- Monitor Hadoop cluster performance, identify bottlenecks, and proactively tune system parameters to ensure optimal performance.
- Implement and manage job scheduling, monitoring, and troubleshooting to optimize MapReduce, Spark, and Hive jobs.
- Provide recommendations for hardware upgrades, cluster expansion, and software improvements.
- Troubleshooting and Issue Resolution:
- Investigate and resolve issues related to Hadoop cluster performance, data processing, and storage.
- Collaborate with data engineers and developers to resolve issues related to data workflows, performance, and system configurations.
- Provide incident management and root cause analysis for any system failures or downtime.
- Backup, Recovery, and Disaster Recovery:
- Implement and manage backup and disaster recovery strategies for Hadoop clusters.
- Ensure data availability and recovery in the event of hardware failures or data corruption.
- Periodically test disaster recovery procedures to ensure minimal downtime.
- System Monitoring and Reporting:
- Set up and maintain monitoring tools to track Hadoop cluster health, resource usage, job execution, and other key metrics.
- Generate regular reports on cluster utilization, performance, and any issues to stakeholders.
- Analyze logs and metrics to identify opportunities for optimization and improvement.
- Collaboration and Documentation:
- Work closely with developers, data scientists, and other stakeholders to provide platform support and ensure the Hadoop environment meets business needs.
- Maintain detailed documentation of cluster configurations, processes, procedures, and troubleshooting steps.
- Provide training to team members on Hadoop ecosystem components and best practices.
Required Skills & Qualifications:
- Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related field.
- 3+ years of experience managing and administering Hadoop clusters, including HDFS, YARN, MapReduce, Hive, Pig, HBase, Spark, Flume, Kafka, etc.
- Strong knowledge of Hadoop ecosystem tools and technologies.
- Hands-on experience with Hadoop cluster installation, configuration, and management.
- Experience with Linux/Unix operating systems, shell scripting, and system administration.
- Proficiency in managing Hadoop security using Kerberos, Ranger, or Sentry for access control.
- Familiarity with data ingestion tools like Sqoop, Flume, or Kafka.
- Knowledge of distributed systems, data replication, and failover mechanisms.
- Familiarity with cloud-based Hadoop platforms (e.g., Amazon EMR, Google Dataproc, or Azure HDInsight) is a plus.
- Strong troubleshooting skills, with the ability to quickly diagnose and resolve issues within the Hadoop environment.
- Good understanding of SQL, HiveQL, or Pig Latin for interacting with Hadoop data stores.
- Familiarity with monitoring tools such as Ambari, Cloudera Manager, or open-source alternatives like Ganglia or Nagios.
- Excellent communication skills, both verbal and written, for documentation and cross-team collaboration.
Preferred Skills:
- Hadoop Certification (e.g., Cloudera Certified Administrator for Apache Hadoop) is a plus.
- Experience with Hadoop ecosystem performance tuning and optimizations for large-scale data processing.
- Familiarity with Apache Spark, Apache Kafka, or other distributed data processing frameworks.
- Experience working with NoSQL databases like HBase or Cassandra in a Hadoop ecosystem.
- Knowledge of containerization (e.g., Docker) and orchestration tools (e.g., Kubernetes) for deploying Hadoop clusters.
