首页 > 代码库 > hadoop测试
hadoop测试
lk@lk-virtual-machine:~$ cd hadoop-1.0.1
lk@lk-virtual-machine:~/hadoop-1.0.1$ cd bin
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./stop_all.sh
bash: ./stop_all.sh: 没有那个文件或目录
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ bin/hadoop namenode -formatbash: bin/hadoop: 没有那个文件或目录
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop namenode -format
14/05/05 01:03:03 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = lk-virtual-machine/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by ‘hortonfo‘ on Tue Feb 14 08:15:38 UTC 2012
************************************************************/
14/05/05 01:03:04 INFO util.GSet: VM type = 32-bit
14/05/05 01:03:04 INFO util.GSet: 2% max memory = 19.33375 MB
14/05/05 01:03:04 INFO util.GSet: capacity = 2^22 = 4194304 entries
14/05/05 01:03:04 INFO util.GSet: recommended=4194304, actual=4194304
14/05/05 01:03:06 INFO namenode.FSNamesystem: fsOwner=lk
14/05/05 01:03:06 INFO namenode.FSNamesystem: supergroup=supergroup
14/05/05 01:03:06 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/05/05 01:03:06 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/05/05 01:03:06 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/05/05 01:03:06 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/05/05 01:03:06 INFO common.Storage: Image file of size 108 saved in 0 seconds.
14/05/05 01:03:07 INFO common.Storage: Storage directory /tmp/hadoop-lk/dfs/name has been successfully formatted.
14/05/05 01:03:07 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at lk-virtual-machine/127.0.1.1
************************************************************/
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./start-all.shstarting namenode, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-namenode-lk-virtual-machine.out
localhost: starting datanode, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-datanode-lk-virtual-machine.out
localhost: starting secondarynamenode, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-secondarynamenode-lk-virtual-machine.out
starting jobtracker, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-jobtracker-lk-virtual-machine.out
localhost: starting tasktracker, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-tasktracker-lk-virtual-machine.out
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop dfs -rmr output
rmr: cannot remove output: No such file or directory.
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -mkdir input
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -put ~/input/file0* input
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -ls inputFound 4 items
-rw-r--r-- 1 lk supergroup 22 2014-05-05 01:08 /user/lk/input/file01
-rw-r--r-- 1 lk supergroup 0 2014-05-05 01:08 /user/lk/input/file01~
-rw-r--r-- 1 lk supergroup 28 2014-05-05 01:08 /user/lk/input/file02
-rw-r--r-- 1 lk supergroup 0 2014-05-05 01:08 /user/lk/input/file02~
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/TopK.jar topk input output
Exception in thread "main" java.lang.ClassNotFoundException: topk
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/TopK.jar topk input output
Exception in thread "main" java.lang.ClassNotFoundException: topk
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/hadoop-1.0.1/hadoop-examples-1.0.1.jar wordcount input output
****hdfs://localhost:9000/user/lk/input
14/05/05 01:15:57 INFO input.FileInputFormat: Total input paths to process : 4
14/05/05 01:16:06 INFO mapred.JobClient: Running job: job_201405050104_0001
14/05/05 01:16:07 INFO mapred.JobClient: map 0% reduce 0%
14/05/05 01:17:24 INFO mapred.JobClient: map 50% reduce 0%
14/05/05 01:18:00 INFO mapred.JobClient: map 100% reduce 0%
14/05/05 01:18:03 INFO mapred.JobClient: map 100% reduce 16%
14/05/05 01:18:09 INFO mapred.JobClient: map 100% reduce 33%
14/05/05 01:18:12 INFO mapred.JobClient: map 100% reduce 100%
14/05/05 01:19:26 INFO mapred.JobClient: Job complete: job_201405050104_0001
14/05/05 01:19:28 INFO mapred.JobClient: Counters: 29
14/05/05 01:19:28 INFO mapred.JobClient: Job Counters
14/05/05 01:19:28 INFO mapred.JobClient: Launched reduce tasks=1
14/05/05 01:19:28 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=194389
14/05/05 01:19:28 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/05 01:19:28 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/05 01:19:28 INFO mapred.JobClient: Launched map tasks=4
14/05/05 01:19:28 INFO mapred.JobClient: Data-local map tasks=2
14/05/05 01:19:28 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=54840
14/05/05 01:19:28 INFO mapred.JobClient: File Output Format Counters
14/05/05 01:19:28 INFO mapred.JobClient: Bytes Written=41
14/05/05 01:19:28 INFO mapred.JobClient: FileSystemCounters
14/05/05 01:19:28 INFO mapred.JobClient: FILE_BYTES_READ=79
14/05/05 01:19:28 INFO mapred.JobClient: HDFS_BYTES_READ=480
14/05/05 01:19:28 INFO mapred.JobClient: FILE_BYTES_WRITTEN=107645
14/05/05 01:19:28 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=41
14/05/05 01:19:28 INFO mapred.JobClient: File Input Format Counters
14/05/05 01:19:28 INFO mapred.JobClient: Bytes Read=50
14/05/05 01:19:28 INFO mapred.JobClient: Map-Reduce Framework
14/05/05 01:19:28 INFO mapred.JobClient: Map output materialized bytes=97
14/05/05 01:19:28 INFO mapred.JobClient: Map input records=2
14/05/05 01:19:28 INFO mapred.JobClient: Reduce shuffle bytes=97
14/05/05 01:19:28 INFO mapred.JobClient: Spilled Records=12
14/05/05 01:19:28 INFO mapred.JobClient: Map output bytes=82
14/05/05 01:19:28 INFO mapred.JobClient: CPU time spent (ms)=11820
14/05/05 01:19:28 INFO mapred.JobClient: Total committed heap usage (bytes)=718479360
14/05/05 01:19:28 INFO mapred.JobClient: Combine input records=8
14/05/05 01:19:28 INFO mapred.JobClient: SPLIT_RAW_BYTES=430
14/05/05 01:19:28 INFO mapred.JobClient: Reduce input records=6
14/05/05 01:19:28 INFO mapred.JobClient: Reduce input groups=5
14/05/05 01:19:28 INFO mapred.JobClient: Combine output records=6
14/05/05 01:19:28 INFO mapred.JobClient: Physical memory (bytes) snapshot=569315328
14/05/05 01:19:28 INFO mapred.JobClient: Reduce output records=5
14/05/05 01:19:28 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1881337856
14/05/05 01:19:28 INFO mapred.JobClient: Map output records=8
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -ls output
Found 3 items
-rw-r--r-- 1 lk supergroup 0 2014-05-05 01:19 /user/lk/output/_SUCCESS
drwxr-xr-x - lk supergroup 0 2014-05-05 01:16 /user/lk/output/_logs
-rw-r--r-- 1 lk supergroup 41 2014-05-05 01:18 /user/lk/output/part-r-00000
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -cat output/part-r-00000Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -rmr outputDeleted hdfs://localhost:9000/user/lk/output
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/wordcount.jar wordcount input output
Exception in thread "main" java.lang.ClassNotFoundException: wordcount
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$
lk@lk-virtual-machine:~/hadoop-1.0.1$ cd bin
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./stop_all.sh
bash: ./stop_all.sh: 没有那个文件或目录
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ bin/hadoop namenode -formatbash: bin/hadoop: 没有那个文件或目录
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop namenode -format
14/05/05 01:03:03 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = lk-virtual-machine/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1243785; compiled by ‘hortonfo‘ on Tue Feb 14 08:15:38 UTC 2012
************************************************************/
14/05/05 01:03:04 INFO util.GSet: VM type = 32-bit
14/05/05 01:03:04 INFO util.GSet: 2% max memory = 19.33375 MB
14/05/05 01:03:04 INFO util.GSet: capacity = 2^22 = 4194304 entries
14/05/05 01:03:04 INFO util.GSet: recommended=4194304, actual=4194304
14/05/05 01:03:06 INFO namenode.FSNamesystem: fsOwner=lk
14/05/05 01:03:06 INFO namenode.FSNamesystem: supergroup=supergroup
14/05/05 01:03:06 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/05/05 01:03:06 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/05/05 01:03:06 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/05/05 01:03:06 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/05/05 01:03:06 INFO common.Storage: Image file of size 108 saved in 0 seconds.
14/05/05 01:03:07 INFO common.Storage: Storage directory /tmp/hadoop-lk/dfs/name has been successfully formatted.
14/05/05 01:03:07 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at lk-virtual-machine/127.0.1.1
************************************************************/
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./start-all.shstarting namenode, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-namenode-lk-virtual-machine.out
localhost: starting datanode, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-datanode-lk-virtual-machine.out
localhost: starting secondarynamenode, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-secondarynamenode-lk-virtual-machine.out
starting jobtracker, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-jobtracker-lk-virtual-machine.out
localhost: starting tasktracker, logging to /home/lk/hadoop-1.0.1/libexec/../logs/hadoop-lk-tasktracker-lk-virtual-machine.out
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop dfs -rmr output
rmr: cannot remove output: No such file or directory.
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -mkdir input
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -put ~/input/file0* input
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -ls inputFound 4 items
-rw-r--r-- 1 lk supergroup 22 2014-05-05 01:08 /user/lk/input/file01
-rw-r--r-- 1 lk supergroup 0 2014-05-05 01:08 /user/lk/input/file01~
-rw-r--r-- 1 lk supergroup 28 2014-05-05 01:08 /user/lk/input/file02
-rw-r--r-- 1 lk supergroup 0 2014-05-05 01:08 /user/lk/input/file02~
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/TopK.jar topk input output
Exception in thread "main" java.lang.ClassNotFoundException: topk
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/TopK.jar topk input output
Exception in thread "main" java.lang.ClassNotFoundException: topk
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/hadoop-1.0.1/hadoop-examples-1.0.1.jar wordcount input output
****hdfs://localhost:9000/user/lk/input
14/05/05 01:15:57 INFO input.FileInputFormat: Total input paths to process : 4
14/05/05 01:16:06 INFO mapred.JobClient: Running job: job_201405050104_0001
14/05/05 01:16:07 INFO mapred.JobClient: map 0% reduce 0%
14/05/05 01:17:24 INFO mapred.JobClient: map 50% reduce 0%
14/05/05 01:18:00 INFO mapred.JobClient: map 100% reduce 0%
14/05/05 01:18:03 INFO mapred.JobClient: map 100% reduce 16%
14/05/05 01:18:09 INFO mapred.JobClient: map 100% reduce 33%
14/05/05 01:18:12 INFO mapred.JobClient: map 100% reduce 100%
14/05/05 01:19:26 INFO mapred.JobClient: Job complete: job_201405050104_0001
14/05/05 01:19:28 INFO mapred.JobClient: Counters: 29
14/05/05 01:19:28 INFO mapred.JobClient: Job Counters
14/05/05 01:19:28 INFO mapred.JobClient: Launched reduce tasks=1
14/05/05 01:19:28 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=194389
14/05/05 01:19:28 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/05 01:19:28 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/05 01:19:28 INFO mapred.JobClient: Launched map tasks=4
14/05/05 01:19:28 INFO mapred.JobClient: Data-local map tasks=2
14/05/05 01:19:28 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=54840
14/05/05 01:19:28 INFO mapred.JobClient: File Output Format Counters
14/05/05 01:19:28 INFO mapred.JobClient: Bytes Written=41
14/05/05 01:19:28 INFO mapred.JobClient: FileSystemCounters
14/05/05 01:19:28 INFO mapred.JobClient: FILE_BYTES_READ=79
14/05/05 01:19:28 INFO mapred.JobClient: HDFS_BYTES_READ=480
14/05/05 01:19:28 INFO mapred.JobClient: FILE_BYTES_WRITTEN=107645
14/05/05 01:19:28 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=41
14/05/05 01:19:28 INFO mapred.JobClient: File Input Format Counters
14/05/05 01:19:28 INFO mapred.JobClient: Bytes Read=50
14/05/05 01:19:28 INFO mapred.JobClient: Map-Reduce Framework
14/05/05 01:19:28 INFO mapred.JobClient: Map output materialized bytes=97
14/05/05 01:19:28 INFO mapred.JobClient: Map input records=2
14/05/05 01:19:28 INFO mapred.JobClient: Reduce shuffle bytes=97
14/05/05 01:19:28 INFO mapred.JobClient: Spilled Records=12
14/05/05 01:19:28 INFO mapred.JobClient: Map output bytes=82
14/05/05 01:19:28 INFO mapred.JobClient: CPU time spent (ms)=11820
14/05/05 01:19:28 INFO mapred.JobClient: Total committed heap usage (bytes)=718479360
14/05/05 01:19:28 INFO mapred.JobClient: Combine input records=8
14/05/05 01:19:28 INFO mapred.JobClient: SPLIT_RAW_BYTES=430
14/05/05 01:19:28 INFO mapred.JobClient: Reduce input records=6
14/05/05 01:19:28 INFO mapred.JobClient: Reduce input groups=5
14/05/05 01:19:28 INFO mapred.JobClient: Combine output records=6
14/05/05 01:19:28 INFO mapred.JobClient: Physical memory (bytes) snapshot=569315328
14/05/05 01:19:28 INFO mapred.JobClient: Reduce output records=5
14/05/05 01:19:28 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1881337856
14/05/05 01:19:28 INFO mapred.JobClient: Map output records=8
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -ls output
Found 3 items
-rw-r--r-- 1 lk supergroup 0 2014-05-05 01:19 /user/lk/output/_SUCCESS
drwxr-xr-x - lk supergroup 0 2014-05-05 01:16 /user/lk/output/_logs
-rw-r--r-- 1 lk supergroup 41 2014-05-05 01:18 /user/lk/output/part-r-00000
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -cat output/part-r-00000Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -rmr outputDeleted hdfs://localhost:9000/user/lk/output
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ~/hadoop-1.0.1/bin/hadoop jar ~/wordcount.jar wordcount input output
Exception in thread "main" java.lang.ClassNotFoundException: wordcount
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$
声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉： 投诉/举报 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。