首页 > 代码库 > spark 执行诡异问题
spark 执行诡异问题
今天在centos 6.3上搭建了hadoop+hive+spark 系统,在运行
/usr/local/spark-1.0.0/bin/spark-shell
出现找不到hive 相关的类,于是修改了spark-env.sh
在spark_classpath 里添加了 hive 的库路径
export SPARK_CLASSPATH=/usr/local/spark-1.0.0/lib_managed/jars/spark-assembly-1.0.0-hadoop2.4.0.jar:/usr/local/spark-1.0.0/assembly/target/scala-2.10/mysql-connector-java-5.1.25-bin.jar:/usr/local/hive-0.13/lib/*
再次运行
/usr/local/spark-1.0.0/bin/spark-shell 又出现
java.lang.IllegalAccessError: tried to access field org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.conf from class org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator
at org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator.setConf(ProxyUserAuthenticator.java:40)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:365)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:166)
进到 /usr/local/spark-1.0.0/assembly/target/scala-2.10/org/apache/hadoop/hive/ql/security 目录下,把authorization 目录修改成另外一个名字 ,再次运行就成了。 原因应该是类重复了(猜的,呵呵)
本文出自 “决胜千里之外” 博客,请务必保留此出处http://lubing.blog.51cto.com/5293532/1546130
spark 执行诡异问题