首页 > 代码库 > Hadoop单机安装配置过程:

Hadoop单机安装配置过程:

 

1. 首先安装JDK,必须是sun公司的jdk,最好1.6版本以上。

最后java –version 查看成功与否。

注意配置/etc/profile文件,在其后面加上下面几句:

export JAVA_HOME=/usr/local/jdk1.6.0_17    
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre:$PATH    
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

 

2. 安装ssh,免密码登陆。

apt-get install openssh-server 安装ssh

建立 SSH KEY: 

$ ssh-keygen -t rsa -P "" (注意不要多加空格或缺少空格)

中途需要安装者选择保存密钥的文件,按回车使用默认的文件就可以了:

Enter file in which to save the key (/root/.ssh/id_rsa):(按回车)

启用SSH KEY:

$ cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

/etc/init.d/ssh reload

reload ssh

ssh localhost

3. 安装配置单机hadoop 

1)解压到/opt/hadoop 

Java代码 

  1. $ tar zxvf hadoop-0.20.2.tar.gz 
  2. $ sudo mv hadoop-0.20.2 /opt/ 
  3. $ sudo chown -R hadoop:hadoop /opt/hadoop-0.20.2 
  4. $ sudo ln -sf /opt/hadoop-0.20.2 /opt/hadoop 

4. 配置 hadoop-env.sh 
1)在hadoop/conf 里面hadoop-env.sh增加 

Java代码 

  1. export JAVA_HOME=/usr/jdk1.6.0.18 
  2. export HADOOP_HOME=/opt/hadoop 
  3. export PATH=$PATH:/opt/hadoop/bin 

5.配置文件 
1)編輯 /opt/hadoop/conf/core-site.xml 

Java代码 

  1. <configuration> 
  2. <property> 
  3. <name>fs.default.name</name> 
  4. <value>hdfs://localhost:9000</value> 
  5. </property> 
  6. <property> 
  7. <name>hadoop.tmp.dir</name> 
  8. <value>/tmp/hadoop/hadoop-${user.name}</value> 
  9. </property> 
  10. </configuration> 

2)編輯 /opt/hadoop/conf/hdfs-site.xml 

Java代码 

  1. <configuration> 
  2. <property> 
  3. <name>dfs.replication</name> 
  4. <value>1</value> 
  5. </property> 
  6. </configuration> 

3)編輯 /opt/hadoop/conf/mapred-site.xml 

Java代码 

  1. <configuration> 
  2. <property> 
  3. <name>mapred.job.tracker</name> 
  4. <value>localhost:9001</value> 
  5. </property> 
  6. </configuration> 

6.格式化hdfs 

Java代码 

  1. $ cd /opt/hadoop 
  2. $ source conf/hadoop-env.sh 
  3. $ hadoop namenode -format 

提示信息一大堆... 

7. 启动hadoop 

Java代码 

  1. $ sudo ./start-all.sh //hadoop/bin下执行 



8.完成后的测试 

Java代码 

  1. http://localhost:50030/ - Hadoop 管理接口 

 

启动
[hadoop@hadoop00 ~]$ ~/hadoop-0.21.0/bin/start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh
starting namenode, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-namenode-hadoop00.out
192.168.91.11: starting datanode, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-datanode-hadoop01.out
192.168.91.12: starting datanode, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-datanode-hadoop02.out
192.168.91.10: starting secondarynamenode, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-secondarynamenode-hadoop00.out
starting jobtracker, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-jobtracker-hadoop00.out
192.168.91.12: starting tasktracker, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-tasktracker-hadoop02.out
192.168.91.11: starting tasktracker, logging to /home/hadoop/hadoop-0.21.0/bin/../logs/hadoop-hadoop-tasktracker-hadoop01.out

2.停止
[hadoop@hadoop00 ~]$ ~/hadoop-0.21.0/bin/stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-mapred.sh
stopping namenode
192.168.91.12: stopping datanode
192.168.91.11: stopping datanode
192.168.91.10: stopping secondarynamenode
stopping jobtracker
192.168.91.11: stopping tasktracker
192.168.91.12: stopping tasktracker

初始配置HDFS 
1 格式化HDFS文件系统
[hadoop@hadoop00 ~]$ hadoop namenode -format

2 查看HDFS
[hadoop@hadoop00 ~]$ hadoop fs -ls /
11/09/24 07:49:55 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/09/24 07:49:56 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
Found 4 items
drwxr-xr-x   - hadoop supergroup          0 2011-09-22 08:05 /home
drwxr-xr-x   - hadoop supergroup          0 2011-09-22 11:29 /jobtracker
drwxr-xr-x   - hadoop supergroup          0 2011-09-22 11:23 /user

3 通过WEB查看hadoop
查看集群状态 http://192.168.91.10:50070/dfshealth.jsp

查看JOB状态 http://192.168.91.10:50030/jobtracker.jsp


运行hadoopexample-wordcount 
Wordcount程序是一个简单的计算输入文件中每个单词出现的次数,并输出到指定的目录下。该程序是官方的例子,在hadoop-0.21.0安装目录下的:hadoop-mapred-examples-0.21.0.jar

hdfs上建立程序的输入目录和文件,同时建立程序的输出目录.
[hadoop@hadoop00 ~]$ mkdir input
[hadoop@hadoop00 ~]$ cat a a a a a b b b c c c c c c c c c 1 1 1 > input/file
[hadoop@hadoop00 ~]$ hadoop fs –mkdir /wordcount
[hadoop@hadoop00 ~]$ hadoop fs –put input /wordcount

[hadoop@hadoop00 ~]$ hadoop jar hadoop-0.21.0/hadoop-mapred-examples-0.21.0.jar wordcount /wordcount/input /wordcount/output
11/09/24 08:11:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/09/24 08:11:26 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/09/24 08:11:26 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/09/24 08:11:26 INFO input.FileInputFormat: Total input paths to process : 2
11/09/24 08:11:26 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
11/09/24 08:11:26 INFO mapreduce.JobSubmitter: number of splits:2
11/09/24 08:11:27 INFO mapreduce.JobSubmitter: adding the following namenodes‘ delegation tokens:null
11/09/24 08:11:27 INFO mapreduce.Job: Running job: job_201109240745_0002
11/09/24 08:11:28 INFO mapreduce.Job:  map 0% reduce 0%
11/09/24 08:11:44 INFO mapreduce.Job:  map 50% reduce 0%
11/09/24 08:11:50 INFO mapreduce.Job:  map 100% reduce 0%
11/09/24 08:11:57 INFO mapreduce.Job:  map 100% reduce 100%
11/09/24 08:11:59 INFO mapreduce.Job: Job complete: job_201109240745_0002
11/09/24 08:11:59 INFO mapreduce.Job: Counters: 34
……

[hadoop@hadoop00 ~]$ hadoop fs -cat /wordcount/output/part-r-00000
11/09/24 08:18:09 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/09/24 08:18:09 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
1       3
a       5
b       3
c       9

 

 

Hadoop单机安装配置过程: