首页 > 代码库 > Storm安装部署
Storm安装部署
1、修改/etc/hosts
172.16.3.7 nimbus
172.16.3.8 supervisor1
172.16.3.9 supervisor2
172.16.3.10 supervisor3
2、集群所有机器安装ZooKeeper,storm需要zk存储数据及进行nimbus、supervisor之间协调
tar xzvf zookeeper-3.4.3.tar.gz
mv zookeeper-3.4.3 ~/platform/zookeeper
cp ~/platform/zookeeper/conf/zoo_sample.cfg ~/platform/zookeeper/conf/zoo.cfg (用zoo_sample.cfg制作$ZOOKEEPER_HOME/conf/zoo.cfg)
配置zoo.cfg:
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home/jjt/platform/zookeeper/data
dataLogDir=/home/jjt/platform/zookeeper/log
# the port at which the clients will connect
clientPort=2181
server.0=nimbus:2888:3888
server.1=supervisor1:2888:3888
server.2=supervisor2:2888:3888
server.3=supervisor3:2888:3888
修改zk id, 每台机器不同，cluster的echo是0，supervisor1的echo是1，supervisor2的echo是2,supervisor3的echo是3
echo 0 > /home/pplive/platform/zookeeper/data/myid
添加环境变量：
export ZOOKEEPER_HOME=/home/pplive/platform/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin
启动zk： zkServer.sh start
3、每台安装zeromq, storm使用zeromq进行消息同学
- # wget http://download.zeromq.org/zeromq-2.2.0.tar.gz
- # tar zxf zeromq-2.2.0.tar.gz
- # cd zeromq-2.2.0
- # ./configure --prefix=/home/jjt/platform/zeromq
- # make
- # make install
- # sudo ldconfig (更新LD_LIBRARY_PATH)
configure中可能会遇到的问题：
configure:error:in ‘/usr/local/download/zeromq-2.2.0‘:
congifure:error:no acceptable C compiler found in $PATH
这是因为没有安装C编译器, 解决方法是：# yum install gcc*
Error:cannot link with -luuid, install uuid-dev
这是因为没有安装uuid相关的package,
解决方法是：# yum install uuid*
# yum install e2fsprogs*
# yum install libuuid*
4、每台安装jzmq, zeromq依赖jzmq
- # https://github.com/nathanmarz/jzmq 下载jzmq-master.zip
- # cd jzmq-master
- # ./autogen.sh
- # ./configure --prefix=/home/pplive/platform/jzmq
- # make
- # make install
./autogen.sh如果报错：autogen.sh:error:could not find libtool is required to run autogen.sh，这是因为缺少了libtool，可以用#yum install libtool*来解决
5、每台安装python，storm启动脚本是python script
- # wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
- # tar zxvf Python-2.7.2.tgz
- # cd Python-2.7.2
- # ./configure
- # make
- # make install
- # vi /etc/ld.so.conf
追加/usr/local/lib/
- # sudo ldconfig
- # Python 查看python是否安装正确
6、每台安装storm
unzip storm-0.8.1.zip
mv storm-0.8.1 ~/platform/storm
修改storm配置 ~/platform/storm/conf/storm.yaml
storm.zookeeper.servers:
- "nimbus"
- "supervisor1"
- "supervisor2"
- "supervisor3"
storm.zookeeper.port: 2181
storm.local.dir: "/home/jjt/platform/storm/data"
java.library.path: /usr/local/lib:/opt/local/lib:/usr/lib:/home/jjt/platform/jzmq/lib:/home/jjt/platform/zeromq/lib?
nimbus.host: "nimbus"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
注意yaml文件key/value之间有一个空格， 当然也可以修改log4j配置log格式
添加环境变量：
export STORM_HOME=/home/pplive/platform/strom
export PATH=$PATH:$STORM_HOME/bin
启动storm:
storm nimubs & (nimbus上)
storm ui & (nimbus上，可通过 http://{NimbusHost}:8080 查看topo运行状态 )
storm supervisor( supervisor上）
至此storm 安装基本完成，可storm启动脚本提供的Option对topo进行控制，比如storm list/kill, 然而在supervisor上执行storm脚本会遇到以下错误：
Exception in thread "main" org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:56)
at backtype.storm.command.list$_main.invoke(list.clj:7)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.command.list.main(Unknown Source)
此时需要在supervisor上将storm配置文件storm.yaml 拷贝至 ~/.storm目录下
由于storm运行中的状态保存在zk中，因此在使用过程中可以重启nimbus/supervisor而不用重启topology, 有时会因为zk中保存有文件相关状态而实际文件已不存在导致nimbus/supervisor启动失败，此时可在zk中将storm相关数据删掉，然后重启zk、storm解决。
Storm安装部署