首页 > 代码库 > 基于HBase Hadoop 分布式集群环境下的MapReduce程序开发

基于HBase Hadoop 分布式集群环境下的MapReduce程序开发

  HBase分布式集群环境搭建成功后,连续4、5天实验客户端Map/Reduce程序开发,这方面的代码网上多得是,写个测试代码非常容易,可是真正运行起来可说是历经挫折。下面就是我最终调通并让程序在集群上运行起来的一些经验教训。

  一、首先说一下我的环境:

  1,集群的环境配置请见这篇博文。

  2,开发客户机环境:操作系统是CentOS6.5,JDK版本是1.7.0-60,开发工具是Eclipse(原始安装是从google的ADT网站下载的ADT专用开发环境,后来加装了Java企业开发的工具,启动Flash还是ADT的:))。

  3,etc/hosts文件添加了集群的所有主机。

  二、必要准备工作:

  1,java类库准备:从集群中scp复制了Hadoop/share下所有文件,hbase/lib目录下的所有文件到开发机的/home/username/lib目录下备用(其中username是开发用户名称),打开eclipse,选择菜单window->Preferences,

在左侧树形菜单中导航至Java--Build Path--User Libraries,我添加了3个User Library,“hadoop-common”,“hadoop-yarn”,“hbase”

然后分别添加如下(这是User Libraries的导出文件,可以在上面的界面中导入)的jar包。

<?xml version="1.0" encoding="UTF-8" standalone="no"?><eclipse-userlibraries version="2">    <library name="hadoop-common" systemlibrary="false">        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/hadoop-auth-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/hadoop-annotations-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/hadoop-common-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-logging-1.1.3.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-compress-1.4.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/guava-11.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-collections-3.2.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/slf4j-api-1.7.5.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/slf4j-log4j12-1.7.5.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/zookeeper-3.4.5-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-configuration-1.6.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-math3-3.1.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/activation-1.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/asm-3.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/avro-1.7.5-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-1.7.0.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-core-1.8.0.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-cli-1.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-codec-1.4.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-digester-1.8.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-el-1.0.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-httpclient-3.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-io-2.4.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-lang-2.6.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-net-3.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/httpclient-4.2.5.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/httpcore-4.2.5.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-core-asl-1.8.8.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-jaxrs-1.8.8.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-xc-1.8.8.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jasper-compiler-5.5.23.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jasper-runtime-5.5.23.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/java-xmlbuilder-0.4.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jaxb-api-2.2.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jaxb-impl-2.2.3-1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-core-1.9.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-json-1.9.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-server-1.9.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jets3t-0.9.0.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jettison-1.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jetty-6.1.26.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jetty-util-6.1.26.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsch-0.1.42.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsp-api-2.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsr305-1.3.9.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/junit-4.8.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/log4j-1.2.17.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/mockito-all-1.8.5.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/paranamer-2.3.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/protobuf-java-2.5.0.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/servlet-api-2.5.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/snappy-java-1.0.4.1.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/stax-api-1.0-2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/xmlenc-0.52.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/xz-1.0.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce/lib/netty-3.6.2.Final.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/hdfs/hadoop-hdfs-2.3.0-cdh5.0.2.jar"/>    </library>    <library name="hadoop-yarn" systemlibrary="false">        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-common-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-shuffle-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-api-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-client-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.0.2.jar"/>    </library>    <library name="hbase" systemlibrary="false">        <archive path="/home/gzg/javadev/lib/hbase/hbase-client-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-common-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-hadoop-compat-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-it-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-protocol-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-testing-util-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-thrift-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/protobuf-java-2.5.0.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-server-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/hbase-shell-0.96.1.1-cdh5.0.2.jar"/>        <archive path="/home/gzg/javadev/lib/hbase/htrace-core-2.01.jar"/>    </library></eclipse-userlibraries>

 注:这个很有可能并不是运行Hbase Map/Reduce任务需要的最精简的类库,恕我没有逐一测试。

  2,新建一个Java/Application,我的名字就叫hbase1,然后在项目project视图中点击右键,选择“Build Path”--“Add Libraries...”--选择“User Library”,然后选中上面添加的三个User Library。

  3,复制集群的三个配置文件到项目的src目录下,这三个配置文件是:

core-site.xml  hdfs-site.xml  yarn-site.xml

   三、编写代码,这里基本按照我所遇挫折过程逐步完善代码(缺少类库之类的就略掉了):

   1,遭遇空指针错误--reverseDNS

主文件内容:

/** * @author gzg * */public class Test1  {	/**	 * @param args	 * @throws IOException 	 * @throws ZooKeeperConnectionException 	 * @throws MasterNotRunningException 	 */	Configuration conf=null;	public static void main(String[] args) {		try {			new TestHBaseMapred().run(false);		} catch(yh.utils.system.CommandError cep){			cep.printStackTrace();		} catch (ClassNotFoundException e) {			// TODO Auto-generated catch block			e.printStackTrace();		} catch (IOException e) {			// TODO Auto-generated catch block			e.printStackTrace();		} catch (InterruptedException e) {			// TODO Auto-generated catch block			e.printStackTrace();		}	}}

 TestHBaseMapRed.java内容:

public class TestHBaseMapred {	public void run(boolean toFile) throws ClassNotFoundException, IOException, InterruptedException{		org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
         //设置zookeeper的地址s conf.set("hbase.zookeeper.quorum", "zk1,zk2,zk3");
         //此设置好像没用 conf.set("hadoop.job.user", "hadoop"); org.apache.hadoop.mapreduce.Job job=null; String tmpTable = "_mrtest1"; yh.utils.ElapsedTimeMeasure etm=yh.utils.ElapsedTimeMeasure.createAndStart(); try { job = Job.getInstance(conf); job.setJobName("count meter"); if(toFile==false){ HBaseUtil.dropTable(conf, tmpTable); HBaseUtil.createTable(conf, tmpTable, "c"); job.setOutputFormatClass(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.class); job.getConfiguration().set(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.OUTPUT_TABLE, tmpTable); // job.setReducerClass(TblReduce.class); }else{ job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class); Path op = new Path("result"); org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.setOutputPath(job, op); // job.setReducerClass(TxtReduce.class); } Scan scan = new Scan(); scan.addFamily(Bytes.toBytes("c")); org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob("dzdl_t1",scan,TblMap.class ,Text.class,LongWritable.class,job); job.waitForCompletion(true); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } System.out.println("time elapsed :"+etm.stop().toString()); } public static class TblMap extends org.apache.hadoop.hbase.mapreduce.TableMapper<Text, LongWritable>{ @Override protected void map(ImmutableBytesWritable key,Result result,org.apache.hadoop.mapreduce.Mapper.Context ctx) throws IOException, InterruptedException{ Cell c=result.getColumnLatestCell(Bytes.toBytes("c"), Bytes.toBytes("col3")); String col= Bytes.toString(CellUtil.cloneQualifier(c)); Text kv = new Text(); kv.set("count_result"); LongWritable lv = new LongWritable(); lv.set(1); ctx.write(kv, lv); //org.apache.log4j.Logger.getRootLogger().info(kv+" : 1"); } } public static class TblReduce extends org.apache.hadoop.hbase.mapreduce.TableReducer<Text,LongWritable,ImmutableBytesWritable>{ @Override protected void reduce(Text key,Iterable<LongWritable> values,Context ctx) throws IOException, InterruptedException{ long t =0; for(LongWritable v:values){ t+=v.get(); } byte[] ok=Bytes.toBytes("count_result"); Put p=new Put(ok); p.add(Bytes.toBytes("c"),key.copyBytes(),Bytes.toBytes(t)); p.add(Bytes.toBytes("c"),Bytes.toBytes("14112896"),Bytes.toBytes(14112896)); ctx.write(new ImmutableBytesWritable(ok),p); } } public static class TxtReduce extends org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer<Text>{ @Override public void reduce(Text key,Iterable<LongWritable> values,Context ctx) throws IOException, InterruptedException{ long t =0; for(LongWritable v:values){ t+=v.get(); } //Put p=new Put(key.copyBytes()); //p.add(Bytes.toBytes("c"),key.copyBytes(),Bytes.toBytes(t)); //ctx.write(new ImmutableBytesWritable(key.copyBytes()),p); LongWritable l = new LongWritable(); l.set(t); ctx.write(key,l); } }}

 此时运行,会报如下错误:

Exception in thread "main" java.lang.NullPointerException	at org.apache.hadoop.net.DNS.reverseDns(DNS.java:92)	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:218)	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:184)	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)	at hbase1.TestHBaseMapred.run(TestHBaseMapred.java:82)	at hbase1.Test1.main(Test1.java:69)

看错误,不知道为什么需要调用DNS服务,hosts文件好像没有作用, 搜索很久,终于在一个老外的问题里找到答案,conf添加如下设置解决。

conf.set("hbase.nameserver.address","nn1,nn2");

 添加后,运行仍会报下面的错无,但是好像并不影响程序继续运行:

2014-08-21 14:05:04,158 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2: Name or service not known]; remaining name ‘111.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,170 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘116.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,171 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘113.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,172 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘114.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,173 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘114.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,173 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘111.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,174 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘116.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,174 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘114.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,175 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘115.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,175 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘116.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,199 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘115.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,200 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘112.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,201 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘115.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,201 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘112.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,202 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘113.5.168.192.in-addr.arpa‘2014-08-21 14:05:04,539 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:15

    2,本地运行的问题。解决了上面的问题,程序终于可以运行了,可是我在rm的web界面中却看不到我提交的任务,再次求助bing,终于找到症结所在,需要添加设置如下:

                //此项必须设置,否则程序只是在本地客户端上运行,通过rm的浏览器管理界面看不到!		conf.set("mapreduce.framework.name", "yarn");		

    3,添加了上面语句后,新的问题又来了,运行程序出现新的错误:

2014-08-21 14:15:08,887 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://rm2:23188/proxy/application_1408378794442_0014/2014-08-21 14:15:08,888 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_1408378794442_00142014-08-21 14:15:21,466 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0014 running in uber mode : false2014-08-21 14:15:21,469 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%2014-08-21 14:15:34,988 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_0, Status : FAILED2014-08-21 14:15:35,023 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000011_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 moreError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:39,485 INFO [org.apache.hadoop.mapreduce.Job] -  map 7% reduce 0%2014-08-21 14:15:39,490 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:39,497 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:39,505 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000001_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:40,663 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%2014-08-21 14:15:40,736 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:42,783 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:44,921 INFO [org.apache.hadoop.mapreduce.Job] -  map 7% reduce 0%2014-08-21 14:15:44,925 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000005_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 moreContainer killed by the ApplicationMaster.Container killed on request. Exit code is 143Container exited with a non-zero exit code 1432014-08-21 14:15:46,019 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%2014-08-21 14:15:46,066 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:46,070 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000002_0, Status : FAILEDContainer killed by the ApplicationMaster.Container killed on request. Exit code is 143Container exited with a non-zero exit code 143Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:46,073 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000009_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:46,076 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000014_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:47,086 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000007_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:15:59,146 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:00,237 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:01,312 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_2, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:01,329 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:03,396 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000008_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:03,401 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:04,488 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000009_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:04,493 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000001_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:05,530 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000010_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:06,546 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000012_0, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:09,672 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_2, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:10,935 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000014_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:19,239 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000012_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:19,243 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_2, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:19,246 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_2, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:20,307 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_2, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:21,326 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000011_1, Status : FAILEDError: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)	at java.security.AccessController.doPrivileged(Native Method)	at javax.security.auth.Subject.doAs(Subject.java:415)	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)	... 8 more2014-08-21 14:16:23,352 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 100%2014-08-21 14:16:24,378 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0014 failed with state FAILED due to: Task failed task_1408378794442_0014_m_000013Job failed as tasks failed. failedMaps:1 failedReduces:02014-08-21 14:16:24,584 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 13	Job Counters 		Failed map tasks=31		Killed map tasks=13		Launched map tasks=44		Other local map tasks=29		Data-local map tasks=15		Total time spent by all maps in occupied slots (ms)=1665094		Total time spent by all reduces in occupied slots (ms)=0		Total time spent by all map tasks (ms)=832547		Total vcore-seconds taken by all map tasks=832547		Total megabyte-seconds taken by all map tasks=1278792192	Map-Reduce Framework		CPU time spent (ms)=0		Physical memory (bytes) snapshot=0		Virtual memory (bytes) snapshot=0*** measure elapsed time,stoped at 2014-08-21 14:16:24:588time elapsed :0days 0hours 1minutes 29seconds 879milliseconds[89.879seconds]

 试着设置 job.setJarByClass(Test1.class); 无效。此问题困扰了我两天,难道只能上传到集群集群上运行吗?这样也太不方便了,N久后,终于我有幸找到了小鱼儿的博客 的这篇文章MapReduce提交作业常见问题,终于是找到了解决方法。我采用的是文中所提第三种方法,在程序开始运行前自己首先打包自己,就是调用的是写好的shell文件,这里也贴出来,以下两个文件在和src同级的dist目录中

[gzg@centos dist]$ cat manifest.mfManifest-Version: 1.0Class-Path: . $HADOOP_HOME/lib hbase1_lib/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.ja r hbase1_lib/hadoop-auth-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-annotat ions-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-common-2.3.0-cdh5.0.2.jar h base1_lib/commons-logging-1.1.3.jar hbase1_lib/commons-compress-1.4.1 .jar hbase1_lib/guava-11.0.2.jar hbase1_lib/commons-collections-3.2.1 .jar hbase1_lib/slf4j-api-1.7.5.jar hbase1_lib/slf4j-log4j12-1.7.5.ja r hbase1_lib/zookeeper-3.4.5-cdh5.0.2.jar hbase1_lib/commons-configur ation-1.6.jar hbase1_lib/commons-math3-3.1.1.jar hbase1_lib/activatio n-1.1.jar hbase1_lib/asm-3.2.jar hbase1_lib/avro-1.7.5-cdh5.0.2.jar h base1_lib/commons-beanutils-1.7.0.jar hbase1_lib/commons-beanutils-co re-1.8.0.jar hbase1_lib/commons-cli-1.2.jar hbase1_lib/commons-codec- 1.4.jar hbase1_lib/commons-digester-1.8.jar hbase1_lib/commons-el-1.0 .jar hbase1_lib/commons-httpclient-3.1.jar hbase1_lib/commons-io-2.4. jar hbase1_lib/commons-lang-2.6.jar hbase1_lib/commons-net-3.1.jar hb ase1_lib/httpclient-4.2.5.jar hbase1_lib/httpcore-4.2.5.jar hbase1_li b/jackson-core-asl-1.8.8.jar hbase1_lib/jackson-jaxrs-1.8.8.jar hbase 1_lib/jackson-mapper-asl-1.8.8.jar hbase1_lib/jackson-xc-1.8.8.jar hb ase1_lib/jasper-compiler-5.5.23.jar hbase1_lib/jasper-runtime-5.5.23. jar hbase1_lib/java-xmlbuilder-0.4.jar hbase1_lib/jaxb-api-2.2.2.jar  hbase1_lib/jaxb-impl-2.2.3-1.jar hbase1_lib/jersey-core-1.9.jar hbase 1_lib/jersey-json-1.9.jar hbase1_lib/jersey-server-1.9.jar hbase1_lib /jets3t-0.9.0.jar hbase1_lib/jettison-1.1.jar hbase1_lib/jetty-6.1.26 .jar hbase1_lib/jetty-util-6.1.26.jar hbase1_lib/jsch-0.1.42.jar hbas e1_lib/jsp-api-2.1.jar hbase1_lib/jsr305-1.3.9.jar hbase1_lib/junit-4 .8.2.jar hbase1_lib/log4j-1.2.17.jar hbase1_lib/mockito-all-1.8.5.jar  hbase1_lib/paranamer-2.3.jar hbase1_lib/protobuf-java-2.5.0.jar hbas e1_lib/servlet-api-2.5.jar hbase1_lib/snappy-java-1.0.4.1.jar hbase1_ lib/stax-api-1.0-2.jar hbase1_lib/xmlenc-0.52.jar hbase1_lib/xz-1.0.j ar hbase1_lib/netty-3.6.2.Final.jar hbase1_lib/hadoop-hdfs-2.3.0-cdh5 .0.2.jar hbase1_lib/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2_2.jar  hbase1_lib/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar hbase1_lib /hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar hbase1_lib/hado op-mapreduce-client-common-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-mapre duce-client-shuffle-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-api-2.3 .0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-client-2.3.0-cdh5.0.2.jar hbas e1_lib/hadoop-yarn-common-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-a pplications-distributedshell-2.3.0-cdh5.0.2.jar hbase1_lib/hbase-clie nt-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-common-0.96.1.1-cdh5.0.2.ja r hbase1_lib/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar hbase1_lib/hb ase-hadoop-compat-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-it-0.96.1.1- cdh5.0.2.jar hbase1_lib/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar hbase 1_lib/hbase-protocol-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-shell-0.9 6.1.1-cdh5.0.2.jar hbase1_lib/hbase-testing-util-0.96.1.1-cdh5.0.2.ja r hbase1_lib/hbase-thrift-0.96.1.1-cdh5.0.2.jar hbase1_lib/protobuf-j ava-2.5.0_2.jar hbase1_lib/htrace-core-2.01.jar hbase1_lib/hbase-serv er-0.96.1.1-cdh5.0.2.jarMain-Class: hbase1.Test1

 

[gzg@centos dist]$ cat makejarcd "$(dirname $0)"rm -f hbase1.jarjar cvfm hbase1.jar manifest.mf  -C "../bin/" .

 同时,修改了TestHBaseMapred.run方法签名,传递jar包路径,新的run方法如下:

public void run(String appJarFile,boolean toFile) throws ClassNotFoundException, IOException, InterruptedException{		//org.apache.hadoop.conf.Configuration conf = yh.utils.hbase.HBaseUtil.getConfiguration("zk1,zk2,zk3");		org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();		conf.set("hbase.zookeeper.quorum", "zk1,zk2,zk3");		conf.set("hadoop.job.user", "hadoop");		conf.set("mapreduce.framework.name", "yarn");				conf.set("hbase.nameserver.address","nn1,nn2");						org.apache.hadoop.mapreduce.Job job=null;		String  tmpTable = "_mrtest1";		yh.utils.ElapsedTimeMeasure etm=yh.utils.ElapsedTimeMeasure.createAndStart();		try {						job = Job.getInstance(conf);						job.setJobName("count meter");			//job.setJarByClass(Test1.class);			job.setJar(appJarFile);						if(toFile==false){				HBaseUtil.dropTable(conf, tmpTable);				HBaseUtil.createTable(conf, tmpTable, "c");				job.setOutputFormatClass(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.class);				job.getConfiguration().set(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.OUTPUT_TABLE, tmpTable);				//				job.setReducerClass(TblReduce.class);			}else{							job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);				Path op = new Path("result");							org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.setOutputPath(job, op);				//				job.setReducerClass(TxtReduce.class);			}									Scan scan = new Scan();			scan.addFamily(Bytes.toBytes("c"));			org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob("dzdl_t1",scan,TblMap.class ,Text.class,LongWritable.class,job);			job.waitForCompletion(true);		} catch (IOException e) {			// TODO Auto-generated catch block			e.printStackTrace();		}		System.out.println("time elapsed :"+etm.stop().toString());			}

    4,在选择写入文件的Reduce方法是,同样遇到权限问题,我采用的解决方法是给所有用户授予了读写权限,这里注意/tmp也要授权,运行时用到

[hadoop@zk1 hbase]$ hdfs dfs -chmod 777 /user[hadoop@zk1 hbase]$ hdfs dfs -chmod 777 /tmp

    另外,最后还有一点注意,设置log4j.properties文件中的级别为INFO,不要设置为WARN,否则看不到很多信息,不利于发现错误原因。

# Configure logging for testing: optionally with log file#log4j.rootLogger=WARN, stdoutlog4j.rootLogger=INFO, stdout# log4j.rootLogger=WARN, stdout, logfilelog4j.appender.stdout=org.apache.log4j.ConsoleAppenderlog4j.appender.stdout.layout=org.apache.log4j.PatternLayoutlog4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%nlog4j.appender.logfile=org.apache.log4j.FileAppenderlog4j.appender.logfile.File=target/spring.loglog4j.appender.logfile.layout=org.apache.log4j.PatternLayoutlog4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

 

    四、At last,终于大功告成,看看map/reduce的运行过程

*** measure elapsed time,started at 2014-08-21 14:48:35:8802014-08-21 14:48:36,601 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2014-08-21 14:48:36,682 INFO [org.apache.hadoop.conf.Configuration.deprecation] - hadoop.native.lib is deprecated. Instead, use io.native.lib.available2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.5-cdh5.0.2--1, built on 06/09/2014 16:09 GMT2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=211.137.170.2462014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_602014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=/usr/java/jdk1.7.0_60/jre2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=/home/gzg/javadev/lib/hadoop/mapreduce/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/hadoop-auth-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/hadoop-annotations-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/hadoop-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-logging-1.1.3.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-compress-1.4.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/guava-11.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-collections-3.2.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/zookeeper-3.4.5-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-configuration-1.6.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-math3-3.1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/activation-1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/asm-3.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/avro-1.7.5-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-cli-1.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-codec-1.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-digester-1.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-el-1.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-httpclient-3.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-io-2.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-lang-2.6.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-net-3.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/httpclient-4.2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/httpcore-4.2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/gzg/javadev/lib/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/gzg/javadev/lib/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-core-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-json-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-server-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jets3t-0.9.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/jettison-1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jetty-6.1.26.jar:/home/gzg/javadev/lib/hadoop/common/lib/jetty-util-6.1.26.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsch-0.1.42.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsp-api-2.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsr305-1.3.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/junit-4.8.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/log4j-1.2.17.jar:/home/gzg/javadev/lib/hadoop/common/lib/mockito-all-1.8.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/paranamer-2.3.jar:/home/gzg/javadev/lib/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/servlet-api-2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/stax-api-1.0-2.jar:/home/gzg/javadev/lib/hadoop/common/lib/xmlenc-0.52.jar:/home/gzg/javadev/lib/hadoop/common/lib/xz-1.0.jar:/home/gzg/javadev/lib/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/gzg/javadev/lib/hadoop/hdfs/hadoop-hdfs-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-shuffle-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-api-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-client-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-client-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-common-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-hadoop-compat-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-it-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-protocol-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-testing-util-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-thrift-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/protobuf-java-2.5.0.jar:/home/gzg/javadev/lib/hbase/hbase-server-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-shell-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/htrace-core-2.01.jar:/home/gzg/javadev/hbase1/bin:/usr/java/jdk1.7.0_60/jre/lib:/usr/lib642014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=/tmp2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA>2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Linux2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd642014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=2.6.32-431.el6.x86_642014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=gzg2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=/home/gzg2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=/home/gzg/javadev/hbase12014-08-21 14:48:42,230 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:42,254 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:42,261 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:42,276 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session2014-08-21 14:48:42,335 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009d, negotiated timeout = 400002014-08-21 14:48:43,407 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:43,408 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:43,409 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk1/192.168.5.121:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:43,412 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk1/192.168.5.121:2181, initiating session2014-08-21 14:48:43,421 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk1/192.168.5.121:2181, sessionid = 0x15478ef8bf2f016f, negotiated timeout = 400002014-08-21 14:48:53,961 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15478ef8bf2f016f closed2014-08-21 14:48:53,961 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:53,985 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Started disable of _mrtest12014-08-21 14:48:54,051 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:54,053 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:54,057 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:54,067 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session2014-08-21 14:48:54,072 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d0, negotiated timeout = 400002014-08-21 14:48:54,119 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d0 closed2014-08-21 14:48:54,119 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:54,235 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:54,236 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:54,237 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:54,243 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session2014-08-21 14:48:54,281 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009e, negotiated timeout = 400002014-08-21 14:48:54,325 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009e closed2014-08-21 14:48:54,326 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:54,558 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:54,560 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:54,561 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:54,562 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session2014-08-21 14:48:54,568 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d1, negotiated timeout = 400002014-08-21 14:48:54,584 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d1 closed2014-08-21 14:48:54,584 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:54,894 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:54,899 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:54,900 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:54,901 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session2014-08-21 14:48:54,906 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d2, negotiated timeout = 400002014-08-21 14:48:54,922 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d2 closed2014-08-21 14:48:54,922 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:55,430 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:55,431 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:55,432 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk1/192.168.5.121:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:55,433 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk1/192.168.5.121:2181, initiating session2014-08-21 14:48:55,439 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk1/192.168.5.121:2181, sessionid = 0x15478ef8bf2f0170, negotiated timeout = 400002014-08-21 14:48:55,456 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15478ef8bf2f0170 closed2014-08-21 14:48:55,457 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:55,459 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Disabled _mrtest12014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Deleted _mrtest12014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing master protocol: MasterService2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1647ed72def6009d2014-08-21 14:48:55,658 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009d closed2014-08-21 14:48:55,659 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:55,824 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=hconnection-0x30dd1e89, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:55,826 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x30dd1e89 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:55,826 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:55,836 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session2014-08-21 14:48:55,855 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009f, negotiated timeout = 400002014-08-21 14:48:56,473 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x30dd1e89, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:56,474 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x30dd1e89 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:56,475 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:56,476 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session2014-08-21 14:48:56,482 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def600a0, negotiated timeout = 400002014-08-21 14:48:56,500 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def600a0 closed2014-08-21 14:48:56,501 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:56,502 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing master protocol: MasterService2014-08-21 14:48:56,502 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1647ed72def6009f2014-08-21 14:48:56,506 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009f closed2014-08-21 14:48:56,507 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down2014-08-21 14:48:56,508 INFO [root] - Table _mrtest1 created!2014-08-21 14:48:56,909 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=90000 watcher=hconnection-0x6a86ad83, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase2014-08-21 14:48:56,911 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)2014-08-21 14:48:56,912 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x6a86ad83 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:21812014-08-21 14:48:56,915 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session2014-08-21 14:48:56,922 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d3, negotiated timeout = 400002014-08-21 14:48:56,965 INFO [org.apache.hadoop.hbase.mapreduce.TableOutputFormat] - Created table instance for _mrtest12014-08-21 14:49:07,111 INFO [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] - Failing over to rm22014-08-21 14:49:12,459 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.2014-08-21 14:49:25,381 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2: Name or service not known]; remaining name ‘111.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,385 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘116.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,386 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘113.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,387 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘114.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,387 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘114.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,388 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘111.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,389 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘116.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,389 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘114.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,390 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘115.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,390 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘116.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,413 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘115.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,414 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘112.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,415 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘115.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,415 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘112.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,416 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name ‘113.5.168.192.in-addr.arpa‘2014-08-21 14:49:25,595 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:152014-08-21 14:49:25,925 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_1408378794442_00152014-08-21 14:49:26,170 INFO [org.apache.hadoop.yarn.client.api.impl.YarnClientImpl] - Submitted application application_1408378794442_00152014-08-21 14:49:26,219 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://rm2:23188/proxy/application_1408378794442_0015/2014-08-21 14:49:26,220 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_1408378794442_00152014-08-21 14:49:38,592 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0015 running in uber mode : false2014-08-21 14:49:38,594 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%2014-08-21 14:50:40,813 INFO [org.apache.hadoop.mapreduce.Job] -  map 7% reduce 0%2014-08-21 14:50:47,911 INFO [org.apache.hadoop.mapreduce.Job] -  map 11% reduce 0%2014-08-21 14:50:49,945 INFO [org.apache.hadoop.mapreduce.Job] -  map 13% reduce 0%2014-08-21 14:51:03,145 INFO [org.apache.hadoop.mapreduce.Job] -  map 20% reduce 0%2014-08-21 14:51:04,159 INFO [org.apache.hadoop.mapreduce.Job] -  map 20% reduce 4%2014-08-21 14:51:10,265 INFO [org.apache.hadoop.mapreduce.Job] -  map 20% reduce 7%2014-08-21 14:51:28,566 INFO [org.apache.hadoop.mapreduce.Job] -  map 24% reduce 7%2014-08-21 14:51:30,598 INFO [org.apache.hadoop.mapreduce.Job] -  map 33% reduce 7%2014-08-21 14:51:31,613 INFO [org.apache.hadoop.mapreduce.Job] -  map 33% reduce 9%2014-08-21 14:51:34,656 INFO [org.apache.hadoop.mapreduce.Job] -  map 33% reduce 11%2014-08-21 14:51:42,840 INFO [org.apache.hadoop.mapreduce.Job] -  map 40% reduce 11%2014-08-21 14:51:46,912 INFO [org.apache.hadoop.mapreduce.Job] -  map 40% reduce 13%2014-08-21 14:51:55,050 INFO [org.apache.hadoop.mapreduce.Job] -  map 47% reduce 13%2014-08-21 14:51:56,074 INFO [org.apache.hadoop.mapreduce.Job] -  map 47% reduce 16%2014-08-21 14:52:03,177 INFO [org.apache.hadoop.mapreduce.Job] -  map 53% reduce 16%2014-08-21 14:52:08,241 INFO [org.apache.hadoop.mapreduce.Job] -  map 53% reduce 18%2014-08-21 14:52:12,303 INFO [org.apache.hadoop.mapreduce.Job] -  map 60% reduce 18%2014-08-21 14:52:14,343 INFO [org.apache.hadoop.mapreduce.Job] -  map 60% reduce 20%2014-08-21 14:52:26,528 INFO [org.apache.hadoop.mapreduce.Job] -  map 64% reduce 20%2014-08-21 14:52:27,551 INFO [org.apache.hadoop.mapreduce.Job] -  map 73% reduce 20%2014-08-21 14:52:29,581 INFO [org.apache.hadoop.mapreduce.Job] -  map 73% reduce 22%2014-08-21 14:52:32,621 INFO [org.apache.hadoop.mapreduce.Job] -  map 73% reduce 24%2014-08-21 14:52:44,811 INFO [org.apache.hadoop.mapreduce.Job] -  map 80% reduce 24%2014-08-21 14:52:47,848 INFO [org.apache.hadoop.mapreduce.Job] -  map 80% reduce 27%2014-08-21 14:52:56,950 INFO [org.apache.hadoop.mapreduce.Job] -  map 84% reduce 27%2014-08-21 14:52:57,966 INFO [org.apache.hadoop.mapreduce.Job] -  map 87% reduce 27%2014-08-21 14:52:59,997 INFO [org.apache.hadoop.mapreduce.Job] -  map 87% reduce 29%2014-08-21 14:53:42,487 INFO [org.apache.hadoop.mapreduce.Job] -  map 91% reduce 29%2014-08-21 14:53:43,506 INFO [org.apache.hadoop.mapreduce.Job] -  map 93% reduce 29%2014-08-21 14:53:45,540 INFO [org.apache.hadoop.mapreduce.Job] -  map 93% reduce 31%2014-08-21 14:53:47,575 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 31%2014-08-21 14:53:48,584 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 33%2014-08-21 14:53:51,611 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 44%2014-08-21 14:53:54,642 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 55%2014-08-21 14:53:57,679 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 66%2014-08-21 14:54:00,710 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 71%2014-08-21 14:54:03,749 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 79%2014-08-21 14:54:06,791 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 88%2014-08-21 14:54:09,829 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 96%2014-08-21 14:54:11,854 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 100%2014-08-21 14:54:11,873 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0015 completed successfully2014-08-21 14:54:12,082 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 61	File System Counters		FILE: Number of bytes read=324596614		FILE: Number of bytes written=651096206		FILE: Number of read operations=0		FILE: Number of large read operations=0		FILE: Number of write operations=0		HDFS: Number of bytes read=1105		HDFS: Number of bytes written=0		HDFS: Number of read operations=15		HDFS: Number of large read operations=0		HDFS: Number of write operations=0	Job Counters 		Killed map tasks=6		Launched map tasks=21		Launched reduce tasks=1		Data-local map tasks=15		Rack-local map tasks=6		Total time spent by all maps in occupied slots (ms)=5299964		Total time spent by all reduces in occupied slots (ms)=413798		Total time spent by all map tasks (ms)=2649982		Total time spent by all reduce tasks (ms)=206899		Total vcore-seconds taken by all map tasks=2649982		Total vcore-seconds taken by all reduce tasks=206899		Total megabyte-seconds taken by all map tasks=4070372352		Total megabyte-seconds taken by all reduce tasks=423729152	Map-Reduce Framework		Map input records=14112896		Map output records=14112896		Map output bytes=296370816		Map output materialized bytes=324596698		Input split bytes=1105		Combine input records=0		Combine output records=0		Reduce input groups=1		Reduce shuffle bytes=324596698		Reduce input records=14112896		Reduce output records=1		Spilled Records=28225792		Shuffled Maps =15		Failed Shuffles=0		Merged Map outputs=15		GC time elapsed (ms)=13168		CPU time spent (ms)=403830		Physical memory (bytes) snapshot=6978674688		Virtual memory (bytes) snapshot=29039456256		Total committed heap usage (bytes)=4911529984	HBase Counters		BYTES_IN_REMOTE_RESULTS=194636060		BYTES_IN_RESULTS=5490524772		MILLIS_BETWEEN_NEXTS=1834495		NOT_SERVING_REGION_EXCEPTION=0		NUM_SCANNER_RESTARTS=0		REGIONS_SCANNED=15		REMOTE_RPC_CALLS=5050		REMOTE_RPC_RETRIES=0		RPC_CALLS=141167		RPC_RETRIES=0	Shuffle Errors		BAD_ID=0		CONNECTION=0		IO_ERROR=0		WRONG_LENGTH=0		WRONG_MAP=0		WRONG_REDUCE=0	File Input Format Counters 		Bytes Read=0	File Output Format Counters 		Bytes Written=0*** measure elapsed time,stoped at 2014-08-21 14:54:12:089time elapsed :0days 0hours 5minutes 36seconds 209milliseconds[336.209seconds]