首页 > 代码库 > CentOS系统下Hadoop 2.4.1集群安装配置（简易版）

CentOS系统下Hadoop 2.4.1集群安装配置（简易版）

2024-07-31 11:24:51   221人阅读

安装配置

1、软件下载

JDK下载:jdk-7u65-linux-i586.tar.gz

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

Hadoop下载：hadoop-2.4.1.tar.gz

http://www.apache.org/dyn/closer.cgi/hadoop/common/

 

2、/etc/hosts配置

 

[html] view plaincopyprint?
 
  1. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4  
  2. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6  
  3.   
  4. <strong><span style="color:#ff0000;">192.168.1.2 Master.Hadoop  
  5. 192.168.1.3 Slave1.Hadoop</span></strong>  

 

3、/etc/profile配置

 

 

[html] view plaincopyprint?
 
  1. export JAVA_HOME=/usr/java/jrockit-jdk1.6.0_45-R28.2.7-4.1.0  
  2. export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib  
  3. export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin  
  4.   
  5. export HADOOP_HOME=/usr/hadoop  
  6. export HADOOP_HOME_WARN_SUPPRESS=1  
  7. export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin  

 

 

4、~/etc/hadoop/core-site.xml配置

 

 

[html] view plaincopyprint?
 
  1. <configuration>  
  2.       
  3.         <property>   
  4.            <name>fs.defaultFS</name>   
  5.            <value>hdfs://Master.Hadoop:9000</value>   
  6.            <description>   
  7.               Where to find the Hadoop Filesystem through the network.   
  8.               Note 9000 is not the default port.   
  9.               (This is slightly changed from previous versions which didnt have "hdfs")   
  10.            </description>   
  11.    </property>   
  12.      
  13.    <property>  
  14.             <name>hadoop.tmp.dir</name>  
  15.             <value>/usr/hadoop/tmp</value>  
  16.     </property>  
  17.       
  18. </configuration>  

 

 

5、~/etc/hadoop/mapred-site.xml配置

 

 

[html] view plaincopyprint?
 
  1. <configuration>      
  2.     <property>   
  3.         <name>mapreduce.framework.name</name>   
  4.     <value>yarn</value>   
  5.     </property>  
  6. </configuration>  

 

 

6、etc/hadoop/yarn-site.xml配置

 

 

[html] view plaincopyprint?
 
  1. <configuration>  
  2.       
  3.     <property>      
  4.     <name>yarn.resourcemanager.scheduler.address</name>    
  5.     <value>Master.Hadoop:8030</value>   
  6.     </property>  
  7.   
  8.     <property>      
  9.         <name>yarn.resourcemanager.resource-tracker.address</name>     
  10.     <value>Master.Hadoop:8031</value>   
  11.     </property>  
  12.       
  13.     <property>    
  14.     <name>yarn.resourcemanager.address</name>       
  15.     <value>Master.Hadoop:8032</value>    
  16.     </property>  
  17.       
  18.     <property>    
  19.     <name>yarn.resourcemanager.admin.address</name>    
  20.     <value>Master.Hadoop:8033</value>     
  21.     </property>   
  22.         
  23.    <property>      
  24.     <name>yarn.resourcemanager.webapp.address</name>      
  25.     <value>Master.Hadoop:8088</value>    
  26.     </property>  
  27.       
  28.     <property>      
  29.     <name>yarn.resourcemanager.webapp.https.address</name>      
  30.     <value>Master.Hadoop:8090</value>    
  31.     </property>  
  32.       
  33.     <property>    
  34.     <name>yarn.nodemanager.local-dirs</name>    
  35.     <value>${hadoop.tmp.dir}/nodemanager/local</value>    
  36.     <description>the local directories used by the nodemanager</description>    
  37.   </property>  
  38.     
  39.   <property>    
  40.     <name>yarn.nodemanager.remote-app-log-dir</name>    
  41.     <value>${hadoop.tmp.dir}/nodemanager/remote</value>    
  42.     <description>directory on hdfs where the application logs are moved to </description>    
  43.   </property>  
  44.     
  45.   <property>    
  46.     <name>yarn.nodemanager.log-dirs</name>    
  47.     <value>${hadoop.tmp.dir}/nodemanager/logs</value>    
  48.     <description>the directories used by Nodemanagers as log directories</description>    
  49.   </property>  
  50.     
  51.   <property>    
  52.     <name>yarn.nodemanager.aux-services</name>    
  53.     <value>mapreduce_shuffle</value>    
  54.     <description>shuffle service that needs to be set for Map Reduce to run </description>    
  55.   </property>  
  56.     
  57.   <property>      
  58.     <name>mapreduce.jobhistory.address</name>     
  59.     <value>Master.Hadoop:10020</value>   
  60.     </property>  
  61.          
  62.     <property>      
  63.     <name>mapreduce.jobhistory.webapp.address</name>     
  64.     <value>Master.Hadoop:19888</value>   
  65.     </property>  
  66.       
  67. </configuration>  

 

 

7、~/etc/hadoop/hdfs-site.xml配置

 

 

[html] view plaincopyprint?
 
  1. <configuration>  
  2.       
  3.     <property>    
  4.    <name>dfs.permissions.superusergroup</name>    
  5.    <value>root</value>   
  6.     </property>  
  7.       
  8.     <property>   
  9.     <name>dfs.replication</name>   
  10.     <value>1</value>   
  11.   </property>   
  12.     
  13. </configuration>  

 

 

启动与验证

 

1、格式化HDFS文件系统

hadoop namenode -format

2、启动Hadoop

启动前关闭集群中所有机器的防火墙

service iptables stop

启动命令

start-all.sh

3、验证Hadoop

方式一：jps

方式二：hadoop dfsadmin -report

CentOS系统下Hadoop 2.4.1集群安装配置（简易版）


联系
我们