首页 > 代码库 > sqoop安装部署(笔记)
sqoop安装部署(笔记)
sqoop是一个把关系型数据库数据抽向hadoop的工具。同时,也支持将hive、pig等查询的结果导入关系型数据库中存储。
由于,笔者部署的hadoop版本是2.2.0,所以sqoop的版本是:sqoop-1.99.3-bin-hadoop200
1、下载sqoop wget http://mirrors.cnnic.cn/apache/sqoop/1.99.3/sqoop-1.99.3-bin-hadoop200.tar.gz
2、解压sqoop安装文件 tar -zxvf sqoop-1.99.3-bin-hadoop200.tar.gz
3、配置sqoop环境变量
export SQOOP_HOME=/home/likehua/sqoop/sqoop
export PATH = $SQOOP_HOME/bin:$PATH
修改sqoop——server关于hadoop共享jar的引用
vi server/conf/catalina.properties
找到common.loader行,把/usr/lib/hadoop/lib/*.jar改成你的hadoop jar 包目录
例如:/home/hadoop/hadoop/share/hadoop/yarn/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/yarn/*.jar,
/home/hadoop/hadoop/share/hadoop/hdfs/*.jar,
/home/hadoop/hadoop/share/hadoop/hdfs/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/mapreduce/*.jar,
/home/hadoop/hadoop/share/hadoop/mapreduce/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/common/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/common/*.jar
vi server/conf/sqoop.properties
找到:mapreduce.configuration.directory行,修改值为你的hadoop配置文件目录
如:/home/likehua/hadoop/hadoop/etc/hadoop
并且替换@LOGDIR@ 和@BASEDIR@ :
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g
然后找到你的数据库jdbc驱动复制到sqoop/lib目录下,如果不存在则创建.
vi /etc/profile
增加以下内容:
export SQOOP_HOME=/home/hadoop/sqoop
export PATH=$PATH:$SQOOP_HOME/bin
export CATALINA_BASE=$SQOOP_HOME/server
export LOGDIR=$SQOOP_HOME/logs/
启动 [root@db12c sqoop]# ./bin/sqoop.sh server start
Sqoop home directory: /home/likehua/sqoop/sqoop
Setting SQOOP_HTTP_PORT: 12000
Setting SQOOP_ADMIN_PORT: 12001
Using CATALINA_OPTS:
Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Using CATALINA_BASE: /home/likehua/sqoop/sqoop/server
Using CATALINA_HOME: /home/likehua/sqoop/sqoop/server
Using CATALINA_TMPDIR: /home/likehua/sqoop/sqoop/server/temp
Using JRE_HOME: /usr/local/jdk1.7.0
Using CLASSPATH: /home/likehua/sqoop/sqoop/server/bin/bootstrap.jar
(sqoop服务端是一个跑在tomcat上的服务程序)
关闭 sqoop server :./bin/sqoop.sh server stop
使用sqoop客户端:
[root@db12c sqoop]# bin/sqoop.sh client
Sqoop home directory: /home/likehua/sqoop/sqoop
Sqoop Shell: Type ‘help‘ or ‘\h‘ for help.
sqoop:000> show version --all#显示版本:show version --all显示连接器:show connector --all创建连接:create connection --cid 1
client version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
server version:
Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
Protocol version:
[1]
sqoop:000>