首页 > 代码库 > spark 执行诡异问题

spark 执行诡异问题

今天在centos 6.3上搭建了hadoop+hive+spark 系统,在运行

/usr/local/spark-1.0.0/bin/spark-shell  


出现找不到hive 相关的类,于是修改了spark-env.sh


在spark_classpath 里添加了  hive 的库路径

export SPARK_CLASSPATH=/usr/local/spark-1.0.0/lib_managed/jars/spark-assembly-1.0.0-hadoop2.4.0.jar:/usr/local/spark-1.0.0/assembly/target/scala-2.10/mysql-connector-java-5.1.25-bin.jar:/usr/local/hive-0.13/lib/*


再次运行 

/usr/local/spark-1.0.0/bin/spark-shell   又出现


java.lang.IllegalAccessError: tried to access field org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.conf from class org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator

at org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator.setConf(ProxyUserAuthenticator.java:40)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:365)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)

at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:166)



进到 /usr/local/spark-1.0.0/assembly/target/scala-2.10/org/apache/hadoop/hive/ql/security 目录下,把authorization 目录修改成另外一个名字 ,再次运行就成了。 原因应该是类重复了(猜的,呵呵)

本文出自 “决胜千里之外” 博客,请务必保留此出处http://lubing.blog.51cto.com/5293532/1546130

spark 执行诡异问题