首页 > 代码库 > Shark0.9.0安装

Shark0.9.0安装

2024-07-04 11:04:38   219人阅读

折腾了大半天。

1、环境需求：git，java1.7，Hadoop2.2.0，Spark0.9.0，hive-0.11-shark-0.9.0.tar.gz，Shark0.9.0

     这里主要看后两个安装，前几个最好版本不能低于给定的。

2、安装过程：

①、安装hive:

wget https://github.com/amplab/hive/archive/v0.11-shark-0.9.0.tar.gz
mv v0.11-shark-0.9.0 hive-0.11-shark-0.9.0.tar.gz
tar xvzf hive-0.11-shark-0.9.0.tar.gz
mkdir -p /hadoop/hiveWareHouse/
mv hive-env.sh.template hive-env.sh
mv hive-site.xml.template hive-site.xml

在hive-site.xml中添加：

<configuration>
<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/hadoop/hiveWareHouse</value>
    <description>location of default database for the warehouse</description>
</property>
</configuration>

②、安装Shark0.9.0：

0.9.0没有提供binary包，这里采用sbt编译github上的Shark0.9.0项目

git clone https://github.com/amplab/shark.git -b branch-0.9.0 shark-0.9.0
cd shark-0.9.0
SPARK_HADOOP_VERSION=2.2.0  SPARK_YARN=true
./sbt/sbt -m 512
>package

整个编译过程由于需要下载依赖包，所以要持续相当长的时间。
④、配置Shark：

cd ~/shark-0.9.0/conf
mv shark-env.sh.template shark-env.sh
mv log4j.properties.template log4j.properties
vim shark-env.sh

添加如下配置信息：

export SHARK_MASTER_MEM=512m
export SPARK_MEM=512m
export SCALA_HOME=/usr/scala/scala-2.10.3
export HIVE_CONF_DIR=/home/shirdrn/cloud/programs/hive-0.11-shark-0.9.0/conf

⑤、测试：

./bin/shark
CREATE DATABASE user_db;
CREATE TABLE users (login STRING, password STRING, id INT, group INT, user STRING, home STRING, cmd STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘:‘ LINES TERMINATED BY ‘\n‘;
LOAD DATA LOCAL INPATH ‘/etc/passwd‘ INTO TABLE users;
select  * from user;

⑥、Shark0.9.0遇到的错误：

Starting the Shark Command Line Client
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/hive/cli/CliDriver : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: shark.SharkCliDriver.  Program will exit.

解决方法：将java升级至1.7


联系
我们