首页 > 代码库 > 1Nginx+fastdfs分布式文件存储

1Nginx+fastdfs分布式文件存储



准备,将所需的软件传到服务器上,服务器的列表如下:

fastdfs-nginx-module_v1.15.tar.gz

FastDFS_v4.06.tar.gz

libevent-2.0.21-stable.tar.gz

nginx-1.5.6.tar.gz

openssl-1.0.1c.tar.gz

pcre-8.36.tar.gz

zlib-1.2.7.tar.gz

  1. 首先切换到root用户,命令是:

su root

  1. 删除系统自带的低版本的libevent , 注意:这里的trackerstorage端都做相同的操作。

查看系统上是否安装了libevent

rpm -qa | grep libevent

libevent-1.4.13-4.el6.x86_64

删除已经安装过的libevent

yum remove libevent-1.4.13-4.el6.x86_64

3 安装libevent2.0.21,可以通过.wget的方式进行下来所需要的tar

1>.wget https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz

2> tar -zxvf libevent-2.0.21-stable.tar.gz

3> cd libevent-2.0.21-stable

4> ./configure --prefix=/usr/local/libevent

5> make && make install

6>libevent创建软链接到/lib库下,64位系统对应/lib64 (注意下面的的只选择其一)

     ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib/libevent-2.0.so.5

     ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib64/libevent-2.0.so.5

4安装Fastdfs

1>下载Fastdfs,需要翻墙来。

        wget http://fastdfs.googlecode.com/files/FastDFS_v4.06.tar.gz

2> tar -zxvf FastDFS_v4.06.tar.gz

3> cd FastDFS

4>由于定义/usr/local/fastdfsfastdfs安装目录,所以需要修改make.sh

  vim make.sh

  #内容更改如下

  TARGET_PREFIX=/usr/local 修改为 /usr/local/fastdfs

  /etc/fdfs全部替换为 /usr/local/fastdfs/conf

技术分享

vim替换命令,进入末行模式(ESC 再按shift :)

%s/\/etc\/fdfs/\/usr\/local\/fastdfs\/conf/g (注意,这里”\/”表示转义’/’ ,g表示全部替换)

5>安装

./make.sh C_INCLUDE_PATH=/usr/local/libevent/include LIBRARY_PATH=/usr/local/libevent/lib

./make.sh install

===================以上是TrackerStorage都需要安装的部分======================

5配置Tracker(它是单独一台电脑上的,并且在使用storage服务器的时候,此端要先启动)

注意Tracker服务器IP地址不能是127.0.0.1Tacker不能和Storage在一台电脑上。

1>创建tracker目录保存运行日志

        mkdir -m 777 -p /home/fastdfs/tracker

2>修改tracker.conf配置

        vim /usr/local/fastdfs/conf/tracker.conf

3>修改内容如下所示

        # the tracker server port

        port=22122

        # the base path to store data and log files

        base_path=/home/yuqing/fastdfs -> base_path=/home/fastdfs/tracker #日志目录

        #开启自定义server ID取代ip形式,方便内部网络服务器更换ip#**此方式要重点理解,4.0以后新特性

        use_storage_id = true #使用server ID作为storage server标识

        storage_ids_filename = storage_ids.conf #<id> <group_name> <ip_or_hostname>

        id_type_in_filename = id #文件名反解析中包含server ID,以前是ip

复制代码

 

4>移动storage_ids.conf文件

        cp  <FastDFS源码目录>/conf/storage_ids.conf /usr/local/fastdfs/conf/

 

5>编辑storage服务器IDIP地址的对应关系

        vim /usr/local/fastdfs/conf/storage_ids.conf

           修改内容如下所示

        #<id> <group_name> <ip_or_hostname>

         100001          group1          192.168.1.12

        #100002          group2           192.168.1.13

        #100003          group2           192.168.1.14

 

6>编辑tracker服务器启动脚本

        vim /etc/init.d/fdfs_trackerd

        #启动脚本内容如下

 

        #!/bin/bash

        #

        # fdfs_trackerd Starts fdfs_trackerd

        #

        #

        # chkconfig: 2345 99 01

        # description: FastDFS tracker server

        ### BEGIN INIT INFO

        # Provides: $fdfs_trackerd

        ### END INIT INFO

        # Source function library.

        . /etc/init.d/functions

        FastDfs=‘/usr/local/fastdfs‘

        CONF="$FastDfs/conf/tracker.conf"

        if [ ! -f $CONF ]; then

          echo "file $CONF does not exist!"

          exit 2

        fi

        PRG="$FastDfs/bin/fdfs_trackerd"

        if [ ! -f $PRG ]; then

          echo "file $PRG does not exist!"

          exit 2

        fi

        Stop="$FastDfs/bin/stop.sh"

        if [ ! -f $Stop ]; then

          echo "file $Stop does not exist!"

          exit 2

        fi

        Restart="$FastDfs/bin/restart.sh"

        if [ ! -f $Restart ]; then

          echo "file $Restart does not exist!"

          exit 2

        fi

        RETVAL=0

        start() {

            echo -n $"Starting FastDFS tracker server: "

            $PRG $CONF &

            RETVAL=$?

            echo

            return $RETVAL

        }

        stop() {

            echo -n $"Stop FastDFS tracker server: "

            $Stop $PRG $CONF

            RETVAL=$?

            return $RETVAL

        }

        rhstatus() {

            status fdfs_trackerd

        }

        restart() {

                  $Restart $PRG $CONF &

        }

        case "$1" in

          start)

            start

            ;;

          stop)

            stop

            ;;

          status)

            rhstatus

            ;;

          restart|reload)

            restart

            ;;

          condrestart)

            restart

            ;;

          *)

            echo $"Usage: $0 {start|stop|status|restart|condrestart}"

            exit 1

        esac

        exit $?

7>给启动脚本增加权限

        chmod 777 /etc/init.d/fdfs_trackerd

 

8>启动tracker

        service fdfs_trackerd restart

9>启动成功,加入开机启动

        vim /etc/rc.d/rc.local

        #加入内容如下

        service fdfs_trackerd start

10>防火墙开启tracker端口22122

        vim /etc/sysconfig/iptables

        #加入内容如下

        -A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT

11>重启防火墙

        service iptables restart

注意的时候,最后运行Tracker的时候,要关闭防火墙,也就是说要执行service iptables stop

===============================================================================

6.配置storage服务器,注意的是storageNginx是在另外一台电脑上的。

1>创建Storage目录保存运行日志及其data数据

        mkdir-m 777 -p /home/fastdfs/storage

2>修改storage.conf配置

        vim/usr/local/fastdfs/conf/storage.conf

        #修改内容如下所示

 

        #the name of the group this storage server belongs to

        group_name=group1#设置组名

        #the name of the group this storage server belongs to

        #the storage server port #the storage server port

        port=23000

        #the base path to store data and log files #日志目录

        base_path=/home/yuqing/fastdfs-> /home/fastdfs/storage

        #store_path#, based 0, if store_path0 not exists, it‘s value is base_path #data数据存储目录

        #the paths must be exist

        store_path0=/home/fastdfs/storage

        #tracker_server can ocur more than once, and tracker_server format is

        #  "host:port", host can be hostnameor ip address

        tracker_server=192.168.209.121:22122->192.168.1.11:22122  这里192.168.1.11:22122表示的是tracker所在电脑的ip地址

 

 3>编辑启动脚本

        vim/etc/init.d/fdfs_storaged

        #启动脚本内容如下

 

        #!/bin/bash

        #

        #fdfs_storaged Starts fdfs_storaged

        #

        #

        #chkconfig: 2345 99 01

        #description: FastDFS storage server

        ###BEGIN INIT INFO

        #Provides: $fdfs_storaged

        ###END INIT INFO

        #Source function library.

        ./etc/init.d/functions

        FastDfs=‘/usr/local/fastdfs‘

        CONF="$FastDfs/conf/storage.conf"

        if[ ! -f $CONF ]; then

          echo "file $CONF does not exist!"

          exit 2

        fi

        PRG="$FastDfs/bin/fdfs_storaged"

        if[ ! -f $PRG ]; then

          echo "file $PRG does not exist!"

          exit 2

        fi

        Stop="$FastDfs/bin/stop.sh"

        if[ ! -f $Stop ]; then

          echo "file $Stop does not exist!"

          exit 2

        fi

        Restart="$FastDfs/bin/restart.sh"

        if[ ! -f $Restart ]; then

          echo "file $Restart does notexist!"

          exit 2

        fi

        RETVAL=0

        start(){

            echo -n $"Starting FastDFS storageserver: "

            $PRG $CONF &

            RETVAL=$?

            echo

            return $RETVAL

        }

        stop(){

            echo -n $"Stop FastDFS storage server:"

            $Stop $PRG $CONF

            RETVAL=$?

            return $RETVAL

        }

        rhstatus(){

            status fdfs_storaged

        }

        restart(){

                  $Restart$PRG $CONF &

        }

        case"$1" in

          start)

            start

            ;;

          stop)

            stop

            ;;

          status)

            rhstatus

            ;;

          restart|reload)

            restart

            ;;

          condrestart)

            restart

            ;;

          *)

            echo $"Usage: $0{start|stop|status|restart|condrestart}"

            exit 1

        esac

        exit$?

 

4>给启动脚本增加权限

        chmod777 /etc/init.d/fdfs_storaged

5>启动storage (注意的是,启动storage的时候要先启动storage

        servicefdfs_storaged restart

        #正常情况,接下来会出现很多mkdirdata path,这是系统在创建数据目录

 

6>启动成功,加入开机启动

        vim/etc/rc.d/rc.local

        #加入内容如下

        servicefdfs_storaged start

 

7.安装nginx(仅正在有Storage的电脑上安装)

 

1>创建nginx日志目录

        mkdir-m 777 -p /home/www/logs

 

2>安装nginx必需的库:zlib-developenssl-devel pcre

     yum -y install zlib-devel openssl-devel

 

3>手动安装pcre

     1> wgetftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.36.tar.gz

     2> tar -zxvf pcre-8.36.tar.gz

     3> cd pcre-8.36

     4> ./configure

     5> make && make install

     6> ln -s /usr/local/lib/libpcre.so.1 /lib

 

4>安装nginx

     1> wget http://nginx.org/download/nginx-1.5.6.tar.gz

     2> tar -zxvf nginx-1.5.6.tar.gz

     3> cd nginx-1.5.6

     4> ./configure --prefix=/usr/local/nginx --with-http_stub_status_module--with-http_ssl_module  (注意:这里的--with-http_ssl_module可以不加)

     5> make && make install

5>检查nginx配置是否正确

      /usr/local/nginx/sbin/nginx -t

     

6>查看nginx编译选项

      /usr/local/nginx/sbin/nginx -V

 

7>编辑启动脚本

      vim /etc/init.d/nginxd

      #启动脚本内容如下

 

        #!/bin/sh

        #

        #nginx - this script starts and stops the nginx daemin

        #

        #description:  Nginx is an HTTP(S) server,HTTP(S) reverse \

        #               proxy and IMAP/POP3 proxy server

        #processname: nginx

        #config:     /usr/local/nginx/conf/nginx.conf

        #pidfile:     /usr/local/nginx/nginx.pid

 

        #Source function library.

        ./etc/rc.d/init.d/functions

 

        #Source networking configuration.

        ./etc/sysconfig/network

 

        #Check that networking is up.

        ["$NETWORKING" = "no" ] && exit 0

 

        #必填

        nginx="/usr/local/nginx/sbin/nginx"

        prog=$(basename$nginx)

        #必填

        NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"

 

        [-f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

 

        lockfile=/var/lock/subsys/nginx

 

        start(){

            [ -x $nginx ] || exit 5

            [ -f $NGINX_CONF_FILE ] || exit 6

            echo -n $"Starting $prog: "

            daemon $nginx -c $NGINX_CONF_FILE

            retval=$?

            echo

            #service php-fpm start

            [ $retval -eq 0 ] && touch$lockfile

            return $retval

        }

        stop(){

            echo -n $"Stopping $prog: "

            $nginx -s stop

            echo_success

            retval=$?

            echo

            #service php-fpm stop

            [ $retval -eq 0 ] && rm -f$lockfile

            return $retval

        }

        restart(){

            stop

            start

        }

        reload(){

            configtest || return $?

            echo -n $"Reloading $prog: "

            $nginx -s reload

            RETVAL=$?

            echo

        }

        force_reload(){

            restart

        }

        configtest(){

          $nginx -t -c $NGINX_CONF_FILE

        }

        version(){

          $nginx -V

        }

        rh_status(){

            status $prog

        }

        rh_status_q(){

            rh_status >/dev/null 2>&1

        }

 

        case"$1" in

            start)

                  rh_status_q&& exit 0

                  $1

                  ;;

            stop)

                  rh_status_q|| exit 0

                  $1

                  ;;

            restart|configtest|version)

                  $1

                  ;;

            reload)

                  rh_status_q|| exit 7

                  $1

                  ;;

            force-reload)

                  force_reload

                  ;;

            status)

                  rh_status

                  ;;

            condrestart|try-restart)

                  rh_status_q|| exit 0

                      ;;

            *)

                  echo$"Usage: $0{start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest|version}"

                  exit2

        esac

 

8>给启动脚本增加权限

        chmod777 /etc/init.d/nginxd

 

9>启动nginx

        servicenginxd restart

        如果启动失败提示找不到库,ldconfig更新库路径

 

10>启动成功,加入开机启动

        vim/etc/rc.d/rc.local

        #加入内容如下

        servicenginxd start

 

 

 

 

8.安装nginx-module模块(仅在有Storage的电脑上)

 

1>下载fastdfs-nginx-modulegoogle被禁了,需要翻墙

        wgethttp://fastdfs.googlecode.com/files/fastdfs-nginx-module_v1.15.tar.gz

2> tar -zxvffastdfs-nginx-module_v1.15.tar.gz;

3>修改插件配置文件,主要是跟安装fastdfs目录相关

        vim<fastdfs-nginx-module源码目录>/src/config

        #修改内容如下

 

        ngx_addon_name=ngx_http_fastdfs_module

        HTTP_MODULES="$HTTP_MODULESngx_http_fastdfs_module"

        NGX_ADDON_SRCS="$NGX_ADDON_SRCS$ngx_addon_dir/ngx_http_fastdfs_module.c"

        CORE_INCS="$CORE_INCS/usr/local/fastdfs/include/fastdfs /usr/local/fastdfs/include/fastcommon/"

        CORE_LIBS="$CORE_LIBS-L/usr/local/fastdfs/lib -lfastcommon -lfdfsclient"

        CFLAGS="$CFLAGS-D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE=‘256*1024‘-DFDFS_MOD_CONF_FILENAME=‘\"/usr/local/fastdfs/conf/mod_fastdfs.conf\"‘"

 

 

4>复制mod_fastdfs.conf/usr/local/fastdfs/conf/目录下

        cp<fastdfs-nginx-module源码目录>/src/mod_fastdfs.conf /usr/local/fastdfs/conf/

 

5>/usr/local/fastdfs/lib加入系统文件/etc/ld.so.conf中(编译时使用的动态链接库)

        vim/etc/ld.so.conf

        #修改内容如下

        /usr/local/fastdfs/lib

 

6>更新库文件缓存ld.so.cache

        ldconfig

7>编译fastdfs-nginx-module模块

     1> cd nginx-1.5.6/

     2> ./configure此处加上nginx之前的编译参数(使用 /usr/local/nginx/sbin/nginx -V命令查看)--add-module=<fastdfs-nginx-module源码目录>/src

     3> make && make install

8>修改mod_fastdfs.conf配置

        vim/usr/local/fastdfs/conf/mod_fastdfs.conf

        #修改内容如下

        #if load FastDFS parameters from tracker server

        #since V1.12

        #default value is false

        load_fdfs_parameters_from_tracker=true

        #FastDFS tracker_server can ocur more than once, and tracker_server format is

        #  "host:port", host can be hostnameor ip address

        #valid only when load_fdfs_parameters_from_tracker is true

        tracker_server=192.168.25.11:22122

        #the port of the local storage server

        #the default value is 23000

        storage_server_port=23000

        #the group name of the local storage server

        group_name=group1#当前storage机器组名

        #if the url / uri including the group name

        #set to false when uri like /M00/00/00/xxx

        #set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx

        #default value is false

        url_have_group_name= true

        #path(disk or mount point) count, default value is 1

        #must same as storage.conf

        store_path_count=1

        #store_path#, based 0, if store_path0 not exists, it‘s value is base_path

        #the paths must be exist

        #must same as storage.conf

        store_path0=/home/fastdfs/storage

        #set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log

        #empty for output to stderr (apache and nginx error_log file)

        log_filename=/home/www/logs/mod_fastdfs.log

 

 

 9.修改nginx.conf配置

     vim /usr/local/nginx/conf/nginx.conf

     #修改内容如下(下面的可以替换原来文件中的所有内容)

 

        worker_processes  2;

        error_log  /home/www/logs/error.log  notice;

        pid        /home/www/logs/nginx.pid;

 

        worker_rlimit_nofile5120;

        events{

            use epoll;

            worker_connections  5120;

        }

 

        http{

            include       mime.types;

            default_type  application/octet-stream;

 

            sendfile        on;

            tcp_nopush     on;

            keepalive_timeout  60;

            tcp_nodelay on;

 

            server {

                  listen       80;

                  server_name  localhost;

                  location/group1/M00 {

                      alias /home/fastdfs/storage/data;

                      ngx_fastdfs_module;

                  }

            }

        }

 

 

10.测试(注意:如果不成功,还要修改client.config中的内容)

1.创建测试文件

echo "hello nginx fastdfs" >test.html

2.查看fastdfs自带测试命令使用方法

/usr/local/fastdfs/bin/fdfs_test -?

3.上传测试文件,通过tracker转存到storage服务器里

/usr/local/fastdfs/bin/fdfs_test/usr/local/fastdfs/conf/client.conf upload test.html

 

得到以下信息

 

tracker_query_storage_store_list_without_group:

        server1. group_name=, ip_addr=172.16.63.13, port=23000

 

group_name=group1, ip_addr=172.16.63.13,port=23000

storage_upload_by_filename

group_name=group1,remote_filename=M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45.html

source ip address: 172.16.63.13

file timestamp=2014-12-18 20:18:51

file size=6

file crc32=1998380320

file url:http://172.16.63.12:8080/group1/M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45.html

storage_upload_slave_by_filename

group_name=group1, remote_filename=M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45_big.html

source ip address: 172.16.63.13

file timestamp=2014-12-18 20:18:51

file size=6

file crc32=1998380320

file url:http://172.16.63.12:8080/group1/M00/00/00/oYYBAFSSxiuABDlLAAAABncc3SA45_big.html

 

4.打开浏览器,查看上传文件

        http://172.16.63.13/group1/M00/00/00/oYYBAFSSwuKAO89DAAAABncc3SA04.html

 

看到"hello nginx fastdfs"则成功

备注:

1>由于没有设置好防火墙,同组groupstorage之间备份数据失败,关闭防火墙:service iptables stop

2>至少有一个storage到主机安装nginx

3> mod_fastdfs.conf此配置文件安装路径

4> tracker服务器ip不能是127.0.0.1

 

 

1Nginx+fastdfs分布式文件存储