首页 > 代码库 > 【hadoop2.6.0】安装+例子运行
【hadoop2.6.0】安装+例子运行
由于下载的是hadoop的最新版,网上各种杂七杂八的东西都不适用。好在官网上说的也够清楚了。如果有人看这篇文章的话,最大的忠告就是看官网。
官网2.6.0的安装教程:http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html
hdfs指令:http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/FileSystemShell.html
注意:2.6.0默认的都是64位系统的,如果用32位的机器总是会出现下面的警告: 这个可以无视,不会影响结果
Java HotSpot(TM) Client VM warning: You have loaded library /home/software/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It‘s highly recommended that you fix the library with ‘execstack -c <libfile>‘, or link it with ‘-z noexecstack‘.
14/12/04 21:52:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1.安装jdk 我写在另一篇文章里了http://www.cnblogs.com/dplearning/p/4140334.html
2. ssh免密码登陆 http://www.cnblogs.com/dplearning/p/4140352.html
3.配置
etc/hadoop/core-site.xml:
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property></configuration>
etc/hadoop/hdfs-site.xml:
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>
4. $ bin/hdfs namenode -format //格式化
5. $ sbin/start-dfs.sh //开启进程
成功的话,jps查看进程应该是
如果没有DataNode 查看一下日志
如果有错误
java.io.IOException: Incompatible clusterIDs in /tmp/hadoop-root/dfs/data: namenode clusterID = CID-2b67ec7b-5edc-4911-bb22-1bb8092a7613; datanode clusterID = CID-aa4ac802-100d-4d29-813d-c6b92dd78f02
那么,应该是/tmp/hadoop-root 文件夹中还有之前残留的文件,全部清空后重新format,重新启动程序应该就好了。
运行例子:
1.先在hdfs上建个文件夹 bin/hdfs dfs -mkdir -p /user/kzy/input
bin/hdfs dfs -mkdir -p /user/kzy/output
2.先上传一些文件:bin/hdfs dfs -put etc/hadoop/ /user/kzy/input 把etc/hadoop文件上传到hdfs的/user/kzy/input中
3.执行指令
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep /user/kzy/input/hadoop /user/kzy/output/o ‘dfs[a-z.]+‘
注意/user/kzy/output/o 是个没有建立过的文件夹,如果用已有的文件夹会有个警告
4.查看结果
bin/hdfs dfs -cat /user/kzy/output/o/*
再运行一些wordcount,官网的在http://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v1.0
运行
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/kzy/input/hadoop /user/kzy/output/wordcount
用
bin/hdfs dfs -cat /user/kzy/output/wordcount/* 看结果
【hadoop2.6.0】安装+例子运行