首页 > 代码库 > 1Nginx+fastdfs分布式文件存储
1Nginx+fastdfs分布式文件存储
准备,将所需的软件传到服务器上,服务器的列表如下:
fastdfs-nginx-module_v1.15.tar.gz FastDFS_v4.06.tar.gz libevent-2.0.21-stable.tar.gz nginx-1.5.6.tar.gz openssl-1.0.1c.tar.gz pcre-8.36.tar.gz zlib-1.2.7.tar.gz |
首先切换到root用户,命令是:
su root |
删除系统自带的低版本的libevent , 注意:这里的tracker和storage端都做相同的操作。
查看系统上是否安装了libevent rpm -qa | grep libevent libevent-1.4.13-4.el6.x86_64 删除已经安装过的libevent yum remove libevent-1.4.13-4.el6.x86_64 |
3 安装libevent2.0.21,可以通过.wget的方式进行下来所需要的tar包
1>.wget https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz 2> tar -zxvf libevent-2.0.21-stable.tar.gz 3> cd libevent-2.0.21-stable 4> ./configure --prefix=/usr/local/libevent 5> make && make install 6>为libevent创建软链接到/lib库下,64位系统对应/lib64 (注意下面的的只选择其一) ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib/libevent-2.0.so.5 ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib64/libevent-2.0.so.5 |
4安装Fastdfs
1>下载Fastdfs,需要翻墙来。 wget http://fastdfs.googlecode.com/files/FastDFS_v4.06.tar.gz 2> tar -zxvf FastDFS_v4.06.tar.gz 3> cd FastDFS 4>由于定义/usr/local/fastdfs为fastdfs安装目录,所以需要修改make.sh vim make.sh #内容更改如下 TARGET_PREFIX=/usr/local 修改为 /usr/local/fastdfs 将/etc/fdfs全部替换为 /usr/local/fastdfs/conf vim替换命令,进入末行模式(ESC 再按shift :) %s/\/etc\/fdfs/\/usr\/local\/fastdfs\/conf/g (注意,这里”\/”表示转义’/’ ,g表示全部替换) 5>安装 ./make.sh C_INCLUDE_PATH=/usr/local/libevent/include LIBRARY_PATH=/usr/local/libevent/lib ./make.sh install |
===================以上是Tracker和Storage都需要安装的部分======================
5配置Tracker(它是单独一台电脑上的,并且在使用storage服务器的时候,此端要先启动)
注意Tracker服务器IP地址不能是127.0.0.1,Tacker不能和Storage在一台电脑上。
1>创建tracker目录保存运行日志 mkdir -m 777 -p /home/fastdfs/tracker 2>修改tracker.conf配置 vim /usr/local/fastdfs/conf/tracker.conf 3>修改内容如下所示 # the tracker server port port=22122 # the base path to store data and log files base_path=/home/yuqing/fastdfs -> base_path=/home/fastdfs/tracker #日志目录 #开启自定义server ID取代ip形式,方便内部网络服务器更换ip#**此方式要重点理解,4.0以后新特性 use_storage_id = true #使用server ID作为storage server标识 storage_ids_filename = storage_ids.conf #<id> <group_name> <ip_or_hostname> id_type_in_filename = id #文件名反解析中包含server ID,以前是ip 复制代码
4>移动storage_ids.conf文件 cp <FastDFS源码目录>/conf/storage_ids.conf /usr/local/fastdfs/conf/
5>编辑storage服务器ID与IP地址的对应关系 vim /usr/local/fastdfs/conf/storage_ids.conf 修改内容如下所示 #<id> <group_name> <ip_or_hostname> 100001 group1 192.168.1.12 #100002 group2 192.168.1.13 #100003 group2 192.168.1.14
6>编辑tracker服务器启动脚本 vim /etc/init.d/fdfs_trackerd #启动脚本内容如下
#!/bin/bash # # fdfs_trackerd Starts fdfs_trackerd # # # chkconfig: 2345 99 01 # description: FastDFS tracker server ### BEGIN INIT INFO # Provides: $fdfs_trackerd ### END INIT INFO # Source function library. . /etc/init.d/functions FastDfs=‘/usr/local/fastdfs‘ CONF="$FastDfs/conf/tracker.conf" if [ ! -f $CONF ]; then echo "file $CONF does not exist!" exit 2 fi PRG="$FastDfs/bin/fdfs_trackerd" if [ ! -f $PRG ]; then echo "file $PRG does not exist!" exit 2 fi Stop="$FastDfs/bin/stop.sh" if [ ! -f $Stop ]; then echo "file $Stop does not exist!" exit 2 fi Restart="$FastDfs/bin/restart.sh" if [ ! -f $Restart ]; then echo "file $Restart does not exist!" exit 2 fi RETVAL=0 start() { echo -n $"Starting FastDFS tracker server: " $PRG $CONF & RETVAL=$? echo return $RETVAL } stop() { echo -n $"Stop FastDFS tracker server: " $Stop $PRG $CONF RETVAL=$? return $RETVAL } rhstatus() { status fdfs_trackerd } restart() { $Restart $PRG $CONF & } case "$1" in start) start ;; stop) stop ;; status) rhstatus ;; restart|reload) restart ;; condrestart) restart ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart}" exit 1 esac exit $? 7>给启动脚本增加权限 chmod 777 /etc/init.d/fdfs_trackerd
8>启动tracker service fdfs_trackerd restart 9>启动成功,加入开机启动 vim /etc/rc.d/rc.local #加入内容如下 service fdfs_trackerd start 10>防火墙开启tracker端口22122 vim /etc/sysconfig/iptables #加入内容如下 -A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT 11>重启防火墙 service iptables restart |
注意的时候,最后运行Tracker的时候,要关闭防火墙,也就是说要执行service iptables stop
===============================================================================
6.配置storage服务器,注意的是storage和Nginx是在另外一台电脑上的。
1>创建Storage目录保存运行日志及其data数据
mkdir-m 777 -p /home/fastdfs/storage
2>修改storage.conf配置
vim/usr/local/fastdfs/conf/storage.conf
#修改内容如下所示
#the name of the group this storage server belongs to
group_name=group1#设置组名
#the name of the group this storage server belongs to
#the storage server port #the storage server port
port=23000
#the base path to store data and log files #日志目录
base_path=/home/yuqing/fastdfs-> /home/fastdfs/storage
#store_path#, based 0, if store_path0 not exists, it‘s value is base_path #data数据存储目录
#the paths must be exist
store_path0=/home/fastdfs/storage
#tracker_server can ocur more than once, and tracker_server format is
# "host:port", host can be hostnameor ip address
tracker_server=192.168.209.121:22122->192.168.1.11:22122 这里192.168.1.11:22122表示的是tracker所在电脑的ip地址
3>编辑启动脚本
vim/etc/init.d/fdfs_storaged
#启动脚本内容如下
#!/bin/bash
#
#fdfs_storaged Starts fdfs_storaged
#
#
#chkconfig: 2345 99 01
#description: FastDFS storage server
###BEGIN INIT INFO
#Provides: $fdfs_storaged
###END INIT INFO
#Source function library.
./etc/init.d/functions
FastDfs=‘/usr/local/fastdfs‘
CONF="$FastDfs/conf/storage.conf"
if[ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi
PRG="$FastDfs/bin/fdfs_storaged"
if[ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi
Stop="$FastDfs/bin/stop.sh"
if[ ! -f $Stop ]; then
echo "file $Stop does not exist!"
exit 2
fi
Restart="$FastDfs/bin/restart.sh"
if[ ! -f $Restart ]; then
echo "file $Restart does notexist!"
exit 2
fi
RETVAL=0
start(){
echo -n $"Starting FastDFS storageserver: "
$PRG $CONF &
RETVAL=$?
echo
return $RETVAL
}
stop(){
echo -n $"Stop FastDFS storage server:"
$Stop $PRG $CONF
RETVAL=$?
return $RETVAL
}
rhstatus(){
status fdfs_storaged
}
restart(){
$Restart$PRG $CONF &
}
case"$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0{start|stop|status|restart|condrestart}"
exit 1
esac
exit$?
4>给启动脚本增加权限
chmod777 /etc/init.d/fdfs_storaged
5>启动storage (注意的是,启动storage的时候要先启动storage)
servicefdfs_storaged restart
#正常情况,接下来会出现很多mkdirdata path,这是系统在创建数据目录
6>启动成功,加入开机启动
vim/etc/rc.d/rc.local
#加入内容如下
servicefdfs_storaged start
7.安装nginx(仅正在有Storage的电脑上安装)
1>创建nginx日志目录
mkdir-m 777 -p /home/www/logs
2>安装nginx必需的库:zlib-developenssl-devel pcre
yum -y install zlib-devel openssl-devel
3>手动安装pcre
1> wgetftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.36.tar.gz
2> tar -zxvf pcre-8.36.tar.gz
3> cd pcre-8.36
4> ./configure
5> make && make install
6> ln -s /usr/local/lib/libpcre.so.1 /lib
4>安装nginx
1> wget http://nginx.org/download/nginx-1.5.6.tar.gz
2> tar -zxvf nginx-1.5.6.tar.gz
3> cd nginx-1.5.6
4> ./configure --prefix=/usr/local/nginx --with-http_stub_status_module--with-http_ssl_module (注意:这里的--with-http_ssl_module可以不加)
5> make && make install
5>检查nginx配置是否正确
/usr/local/nginx/sbin/nginx -t
6>查看nginx编译选项
/usr/local/nginx/sbin/nginx -V
7>编辑启动脚本
vim /etc/init.d/nginxd
#启动脚本内容如下
#!/bin/sh
#
#nginx - this script starts and stops the nginx daemin
#
#description: Nginx is an HTTP(S) server,HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
#processname: nginx
#config: /usr/local/nginx/conf/nginx.conf
#pidfile: /usr/local/nginx/nginx.pid
#Source function library.
./etc/rc.d/init.d/functions
#Source networking configuration.
./etc/sysconfig/network
#Check that networking is up.
["$NETWORKING" = "no" ] && exit 0
#必填
nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename$nginx)
#必填
NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"
[-f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
start(){
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
#service php-fpm start
[ $retval -eq 0 ] && touch$lockfile
return $retval
}
stop(){
echo -n $"Stopping $prog: "
$nginx -s stop
echo_success
retval=$?
echo
#service php-fpm stop
[ $retval -eq 0 ] && rm -f$lockfile
return $retval
}
restart(){
stop
start
}
reload(){
configtest || return $?
echo -n $"Reloading $prog: "
$nginx -s reload
RETVAL=$?
echo
}
force_reload(){
restart
}
configtest(){
$nginx -t -c $NGINX_CONF_FILE
}
version(){
$nginx -V
}
rh_status(){
status $prog
}
rh_status_q(){
rh_status >/dev/null 2>&1
}
case"$1" in
start)
rh_status_q&& exit 0
$1
;;
stop)
rh_status_q|| exit 0
$1
;;
restart|configtest|version)
$1
;;
reload)
rh_status_q|| exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q|| exit 0
;;
*)
echo$"Usage: $0{start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest|version}"
exit2
esac
8>给启动脚本增加权限
chmod777 /etc/init.d/nginxd
9>启动nginx
servicenginxd restart
如果启动失败提示找不到库,ldconfig更新库路径
10>启动成功,加入开机启动
vim/etc/rc.d/rc.local
#加入内容如下
servicenginxd start
8.安装nginx-module模块(仅在有Storage的电脑上)
1>下载fastdfs-nginx-module,google被禁了,需要翻墙
wgethttp://fastdfs.googlecode.com/files/fastdfs-nginx-module_v1.15.tar.gz
2> tar -zxvffastdfs-nginx-module_v1.15.tar.gz;
3>修改插件配置文件,主要是跟安装fastdfs目录相关
vim<fastdfs-nginx-module源码目录>/src/config
#修改内容如下
ngx_addon_name=ngx_http_fastdfs_module
HTTP_MODULES="$HTTP_MODULESngx_http_fastdfs_module"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS$ngx_addon_dir/ngx_http_fastdfs_module.c"
CORE_INCS="$CORE_INCS/usr/local/fastdfs/include/fastdfs /usr/local/fastdfs/include/fastcommon/"
CORE_LIBS="$CORE_LIBS-L/usr/local/fastdfs/lib -lfastcommon -lfdfsclient"
CFLAGS="$CFLAGS-D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE=‘256*1024‘-DFDFS_MOD_CONF_FILENAME=‘\"/usr/local/fastdfs/conf/mod_fastdfs.conf\"‘"
4>复制mod_fastdfs.conf到/usr/local/fastdfs/conf/目录下
cp<fastdfs-nginx-module源码目录>/src/mod_fastdfs.conf /usr/local/fastdfs/conf/
5>将/usr/local/fastdfs/lib加入系统文件/etc/ld.so.conf中(编译时使用的动态链接库)
vim/etc/ld.so.conf
#修改内容如下
/usr/local/fastdfs/lib
6>更新库文件缓存ld.so.cache
ldconfig
7>编译fastdfs-nginx-module模块
1> cd nginx-1.5.6/
2> ./configure此处加上nginx之前的编译参数(使用 /usr/local/nginx/sbin/nginx -V命令查看)--add-module=<fastdfs-nginx-module源码目录>/src
3> make && make install
8>修改mod_fastdfs.conf配置
vim/usr/local/fastdfs/conf/mod_fastdfs.conf
#修改内容如下
#if load FastDFS parameters from tracker server
#since V1.12
#default value is false
load_fdfs_parameters_from_tracker=true
#FastDFS tracker_server can ocur more than once, and tracker_server format is
# "host:port", host can be hostnameor ip address
#valid only when load_fdfs_parameters_from_tracker is true
tracker_server=192.168.25.11:22122
#the port of the local storage server
#the default value is 23000
storage_server_port=23000
#the group name of the local storage server
group_name=group1#当前storage机器组名
#if the url / uri including the group name
#set to false when uri like /M00/00/00/xxx
#set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx
#default value is false
url_have_group_name= true
#path(disk or mount point) count, default value is 1
#must same as storage.conf
store_path_count=1
#store_path#, based 0, if store_path0 not exists, it‘s value is base_path
#the paths must be exist
#must same as storage.conf
store_path0=/home/fastdfs/storage
#set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log
#empty for output to stderr (apache and nginx error_log file)
log_filename=/home/www/logs/mod_fastdfs.log
9.修改nginx.conf配置
vim /usr/local/nginx/conf/nginx.conf
#修改内容如下(下面的可以替换原来文件中的所有内容)
worker_processes 2;
error_log /home/www/logs/error.log notice;
pid /home/www/logs/nginx.pid;
worker_rlimit_nofile5120;
events{
use epoll;
worker_connections 5120;
}
http{
include mime.types;
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
keepalive_timeout 60;
tcp_nodelay on;
server {
listen 80;
server_name localhost;
location/group1/M00 {
alias /home/fastdfs/storage/data;
ngx_fastdfs_module;
}
}
}
10.测试(注意:如果不成功,还要修改client.config中的内容)
1.创建测试文件
echo "hello nginx fastdfs" >test.html
2.查看fastdfs自带测试命令使用方法
/usr/local/fastdfs/bin/fdfs_test -?
3.上传测试文件,通过tracker转存到storage服务器里
/usr/local/fastdfs/bin/fdfs_test/usr/local/fastdfs/conf/client.conf upload test.html
得到以下信息
tracker_query_storage_store_list_without_group:
server1. group_name=, ip_addr=172.16.63.13, port=23000
group_name=group1, ip_addr=172.16.63.13,port=23000
storage_upload_by_filename
group_name=group1,remote_filename=M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45.html
source ip address: 172.16.63.13
file timestamp=2014-12-18 20:18:51
file size=6
file crc32=1998380320
file url:http://172.16.63.12:8080/group1/M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45.html
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45_big.html
source ip address: 172.16.63.13
file timestamp=2014-12-18 20:18:51
file size=6
file crc32=1998380320
file url:http://172.16.63.12:8080/group1/M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45_big.html
4.打开浏览器,查看上传文件
http://172.16.63.13/group1/M00/00/00/oYYBAFSSwuKAO89DAAAABncc3SA04.html
看到"hello nginx fastdfs"则成功
备注:
1>由于没有设置好防火墙,同组group到storage之间备份数据失败,关闭防火墙:service iptables stop
2>至少有一个storage到主机安装nginx
3> mod_fastdfs.conf此配置文件安装路径
4> tracker服务器ip不能是127.0.0.1
1Nginx+fastdfs分布式文件存储