首页 > 代码库 > topk记录
topk记录
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop dfs -rmr output
Deleted hdfs://localhost:9000/user/lk/output
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop jar ~/mytopk.jar top.Top input output
****hdfs://localhost:9000/user/lk/input
14/05/12 05:14:03 INFO input.FileInputFormat: Total input paths to process : 4
14/05/12 05:14:18 INFO mapred.JobClient: Running job: job_201405120333_0004
14/05/12 05:14:20 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:17:32 INFO mapred.JobClient: map 50% reduce 0%
14/05/12 05:17:36 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "/bin/ls": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
... 15 more
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
14/05/12 05:17:58 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:18:10 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "/bin/ls": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
... 15 more
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
14/05/12 05:18:14 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000001_1, Status : FAILED
14/05/12 05:20:01 INFO mapred.JobClient: map 25% reduce 0%
14/05/12 05:20:27 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000000_1, Status : FAILED
attempt_201405120333_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
attempt_201405120333_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly.
14/05/12 05:20:41 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:20:43 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000001_2, Status : FAILED
java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:493)
at java.lang.Integer.parseInt(Integer.java:514)
at top.Top$TopKMapper.map(Top.java:28)
at top.Top$TopKMapper.map(Top.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
14/05/12 05:23:55 INFO mapred.JobClient: map 25% reduce 0%
14/05/12 05:24:06 INFO mapred.JobClient: map 50% reduce 0%
14/05/12 05:24:19 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000000_2, Status : FAILED
attempt_201405120333_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
attempt_201405120333_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly.
14/05/12 05:24:33 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:24:58 INFO mapred.JobClient: Job complete: job_201405120333_0004
14/05/12 05:25:01 INFO mapred.JobClient: Counters: 7
14/05/12 05:25:01 INFO mapred.JobClient: Job Counters
14/05/12 05:25:01 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=979673
14/05/12 05:25:01 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/12 05:25:01 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/12 05:25:01 INFO mapred.JobClient: Launched map tasks=7
14/05/12 05:25:01 INFO mapred.JobClient: Data-local map tasks=7
14/05/12 05:25:01 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
14/05/12 05:25:01 INFO mapred.JobClient: Failed map tasks=1
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$
Deleted hdfs://localhost:9000/user/lk/output
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$ ./hadoop jar ~/mytopk.jar top.Top input output
****hdfs://localhost:9000/user/lk/input
14/05/12 05:14:03 INFO input.FileInputFormat: Total input paths to process : 4
14/05/12 05:14:18 INFO mapred.JobClient: Running job: job_201405120333_0004
14/05/12 05:14:20 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:17:32 INFO mapred.JobClient: map 50% reduce 0%
14/05/12 05:17:36 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "/bin/ls": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
... 15 more
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
14/05/12 05:17:58 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:18:10 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "/bin/ls": java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:488)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:470)
... 15 more
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
at org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
at org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
14/05/12 05:18:14 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000001_1, Status : FAILED
14/05/12 05:20:01 INFO mapred.JobClient: map 25% reduce 0%
14/05/12 05:20:27 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000000_1, Status : FAILED
attempt_201405120333_0004_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
attempt_201405120333_0004_m_000000_1: log4j:WARN Please initialize the log4j system properly.
14/05/12 05:20:41 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:20:43 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000001_2, Status : FAILED
java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:493)
at java.lang.Integer.parseInt(Integer.java:514)
at top.Top$TopKMapper.map(Top.java:28)
at top.Top$TopKMapper.map(Top.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
14/05/12 05:23:55 INFO mapred.JobClient: map 25% reduce 0%
14/05/12 05:24:06 INFO mapred.JobClient: map 50% reduce 0%
14/05/12 05:24:19 INFO mapred.JobClient: Task Id : attempt_201405120333_0004_m_000000_2, Status : FAILED
attempt_201405120333_0004_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
attempt_201405120333_0004_m_000000_2: log4j:WARN Please initialize the log4j system properly.
14/05/12 05:24:33 INFO mapred.JobClient: map 0% reduce 0%
14/05/12 05:24:58 INFO mapred.JobClient: Job complete: job_201405120333_0004
14/05/12 05:25:01 INFO mapred.JobClient: Counters: 7
14/05/12 05:25:01 INFO mapred.JobClient: Job Counters
14/05/12 05:25:01 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=979673
14/05/12 05:25:01 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/12 05:25:01 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/12 05:25:01 INFO mapred.JobClient: Launched map tasks=7
14/05/12 05:25:01 INFO mapred.JobClient: Data-local map tasks=7
14/05/12 05:25:01 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
14/05/12 05:25:01 INFO mapred.JobClient: Failed map tasks=1
lk@lk-virtual-machine:~/hadoop-1.0.1/bin$
声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉： 投诉/举报 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。