首页 > 代码库 > protobuf 2.5.0问题
protobuf 2.5.0问题
线上一个项目需要操作hdfs和hive,在使用时报如下错误:
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses. at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180) at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407) at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749) at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CreateRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:2371) at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193) at $Proxy28.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:193) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at $Proxy29.create(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1325) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1344) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1255) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1212) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:276) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:265) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886)
根据堆栈信息,可以看到是pb的问题,异常由com.google.protobuf.GeneratedMessage类的getUnknownFields方法抛出。
在2.4.0a中这个方法的定义如下:
//@Override (Java 1.6 override semantics, but we must support 1.5) public final UnknownFieldSet getUnknownFields() { return unknownFields; //会返回一个值 }
而在2.5.0中:
//@Override (Java 1.6 override semantics, but we must support 1.5) public UnknownFieldSet getUnknownFields() { throw new UnsupportedOperationException( //在180行可以看到这个方法直接返回一个错误 "This is supposed to be overridden by subclasses."); }
即这个错误是由于pb的版本导致,项目是运行在tomcat容器下面的,在WEB-INF/lib目录下只发现protobuf-java-2.4.0a.jar的包,同时在整个文件系统中也只有这一个pb包,在删除这个pb包之后,错误仍然存在,也就是有可能pb已经打在别的jar包里面了。通过jar tvf,最终发现pb是在hive-exec的包里面:
jar tvf hive-exec-0.13.1.jar |grep protobuf 0 Mon Jun 02 12:50:20 CST 2014 META-INF/maven/com.google.protobuf/ 0 Mon Jun 02 12:50:20 CST 2014 META-INF/maven/com.google.protobuf/protobuf-java/ 141 Mon Jun 02 12:50:20 CST 2014 META-INF/maven/com.google.protobuf/protobuf-java/pom.properties 8375 Mon Jun 02 12:50:20 CST 2014 META-INF/maven/com.google.protobuf/protobuf-java/pom.xml 0 Mon Jun 02 12:50:20 CST 2014 com/google/protobuf/ 1014 Mon Jun 02 12:50:20 CST 2014 com/google/protobuf/AbstractMessage$1.class 30034 Mon Jun 02 12:50:20 CST 2014 com/google/protobuf/AbstractMessage$Builder.class 7979 Mon Jun 02 12:50:20 CST 2014 com/google/protobuf/AbstractMessage.class ...
同时解压jar之后,查看其pom.properties文件,发现是pb是2.5.0版本的
cd /usr/local/src/META-INF/maven/com.google.protobuf/protobuf-java cat pom.properties #Generated by org.apache.felix.bundleplugin #Thu Mar 07 15:48:28 PST 2013 version=2.5.0 groupId=com.google.protobuf artifactId=protobuf-java
尝试更改了hive的pom文件,把pb的依赖设置为2.4.0a时,编译不能通过,即cdh4.6.0是需要使用pb 2.4.x的,而hive0.13.1默认是使用2.5.0的,而项目默认加载了2.5.0的pb,导致会报错。
根据
http://stackoverflow.com/questions/5474765/order-of-loading-jar-files-from-lib-directory
https://issues.apache.org/bugzilla/show_bug.cgi?id=57129
在 tomcat 5-7 jar包是按字母顺序加载的,因此如果要想使protobuf-java-2.4.0a.jar先于hive-exec-0.13.1.jar加载,可以更改protobuf-java-2.4.0a.jar的文件名。
本文出自 “菜光光的博客” 博客,请务必保留此出处http://caiguangguang.blog.51cto.com/1652935/1592804
protobuf 2.5.0问题