首页 > 代码库 > Rsync文件同步
Rsync文件同步
源自《Linux 运维之道》丁一明编著 一书的总结
Rsync(remote sync)是UNIX平台下一款神奇的数据镜像备份软件。根据数据的变化进行差异备份,从而减少流量,提高工作效率。Rsync使用TCP873端口。
在服务器端:
[root@localhost /]# mkdir /rsyncFile
[root@localhost /]# gedit /etc/rsyncd.conf
添加内容为:
#/etc/rsyncd.conf
#设置服务器信息提示文件名称,在该文件中编写提示信息
motd file = /etc/rsyncd.motd
#开启Rsync数据传输日志功能
transfer logging=yes
#设置日志文件名称,可以通过log format参数设置日志格式
log file=/var/log/rsyncd.log
#设置Rsync进程号保存文件名称
pid file=/var/run/rsyncd.pid
#设置锁文件名称
lock file=/var/run/rsync.lock
#设置服务器监听的端口号,默认为873
port=873
#设置服务器所监听的网卡接口的IP地址
address = 192.168.118.253
#设置进行数据传输时所使用的账户名称或ID号,默认为nobody
uid = nobody
gid = nobody
use chroot = yes
#是否允许客户端上传数据,这里设置为只读
read only = yes
//并发数
max connections=10
[common]
comment=Web content
path=/rsyncFile
ignore errors
auth users=tom,jerry
secrets file=/etc/rsyncd.secrets
hosts allow=192.168.118.0/255.255.255.0
hosts deny=*
#客户端请求显示模块列表时,本模块名称是否显示,默认为true
list=false
[root@localhost /]# echo "tom:123">/etc/rsyncd.secrets
[root@localhost /]# echo "jerry:123">/etc/rsyncd.secrets
为了安全最好设置为600
[root@localhost /]# chmod 600 /etc/rsyncd.secrets
[root@localhost /]# echo "welcome to access">/etc/rsyncd.motd
[root@localhost /]# rsync --daemon
[root@localhost /]# echo "/usr/bin/rsync --daemon">/etc/rc.local
[root@localhost /]# iptables -I INPUT -p tcp --dport 873 -j ACCEPT
[root@localhost /]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@localhost /]# touch rsyncFile/test
[root@localhost /]#
[root@test ~]# yum -y install rsync
[root@test ~]# ls
anaconda-ks.cfg install.log install.log.syslog
[root@test ~]# mkdir /test
[root@test ~]# rsync -vzrtopg --progress jerry@192.168.118.253::common /test
welcome to access
Password:
receiving incremental file list
./
test
0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/2)
sent 79 bytes received 165 bytes 97.60 bytes/sec
total size is 0 speedup is 0.00
针对每次都要设置密码很麻烦
[root@test ~]# echo "123">rsync.pass
[root@test ~]# chmod 600 rsync.pass
[root@test ~]# rsync -avz --delete --password-file=rsync.pass jerry@192.168.118.253::common /test
welcome to access
receiving incremental file list
sent 57 bytes received 126 bytes 366.00 bytes/sec
total size is 0 speedup is 0.00
[root@test ~]#
也可以添加脚本,甚至计划任务,下面是添加脚本
[root@test /]# gedit rsync_bak.sh
脚本内容
#!/bin/bash
#backup through rsync
#Data:2014-6-27
export PATH=/bin:/usr/bin:/usr/local/bin
SRC=http://www.mamicode.com/common
DEST=/data
Server=192.168.118.253
User=jerry
#password file must not be other-accessible
Passfile=/root/rsync.pass
#if the DEST direct not found,then create one.
[ ! -d $DEST ] && mkdir $DEST
[ ! -e $Passfile ] && exit 2
rsync -az --delete --password-file=$Passfile ${User}@${Server}::$SRC $DEST
可以执行
[root@test ~]# sh rsync_bak.sh
welcome to access
Rsync+Inotify实现文件自动同步
如果仅使用Rsync进行数据同步,只能满足企业对数据实时性要求不高的环境,即使使用计划任务也仅可以实现定期的数据同步。而且使用Rsync在进行数据同步前需要对所有的文件进行对比,然后进行差异数据同步,然而我们的数据可能只是1TB数据中的1KB数据发生了改变,在不知道什么会发生数据改变的情况下,为了同步1KB的数据,我们需要不停地进行Rsync连接,对比客户端与服务器之前的数据差异,这样是不高效的,需要结合Inotify工具。
Inotify是一种事件通知机制,用户态的应用程序就可以实时监控文件系统的变化,然而Inotify仅是内核提供的一种系统功能,用户要使用该功能还需要安装用户态软件。
先去https://github.com/rvoicilas/inotify-tools下载相应的包。
[root@localhost inotify-tools-master]# sh autogen.sh
libtoolize: putting auxiliary files in `.‘.
libtoolize: linking file `./ltmain.sh‘
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4‘.
libtoolize: linking file `m4/libtool.m4‘
libtoolize: linking file `m4/ltoptions.m4‘
libtoolize: linking file `m4/ltsugar.m4‘
libtoolize: linking file `m4/ltversion.m4‘
libtoolize: linking file `m4/lt~obsolete.m4‘
libtoolize: Consider adding `-I m4‘ to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:16: installing `./config.guess‘
configure.ac:16: installing `./config.sub‘
configure.ac:6: installing `./install-sh‘
configure.ac:6: installing `./missing‘
libinotifytools/src/Makefile.am:26: docdir was already defined in condition TRUE, which includes condition DOXYGEN_ENABLE ...
configure.ac:5: ... `docdir‘ previously defined here
libinotifytools/src/Makefile.am: installing `./depcomp‘
OK, you can run `./configure‘ now.
[root@localhost inotify-tools-master]# ./configure
[root@localhost inotify-tools-master]# make && make install
启动并进行测试:
[root@localhost inotify-tools-master]# mkdir -p /test
[root@localhost inotify-tools-master]# echo "hello">/test/foo
[root@localhost inotify-tools-master]# inotifywait /test/
Setting up watches.
Watches established.
/test/ MODIFY foo
此时再另外的终端输入[root@localhost ~]# echo "try">/test/foo才会有检测到并退出。
可以写一个检测的脚本
[root@localhost ~]# cat monitor.sh
#!/bin/bash
while inotifywait -e modify /var/log/messages
do
if tail -n1 /var/log/messages | grep NetworkManager
then
echo Love
fi
done
Rsync和Inotify的例子:
在192.168.118.253中设置:
[root@localhost ~]# mkdir -p /var/www/001
[root@localhost ~]# chmod 660 /var/www/001
[root@localhost ~]# chown nobody.nobody /var/www/001
[root@localhost ~]# gedit /etc/rsyncd.conf
修改内容如下:
#/etc/rsyncd.conf
#
transfer logging=yes
log file=/var/log/rsyncd.log
pid file=/var/run/rsyncd.pid
lock file=/var/run/rsync.lock
port=873
address = 192.168.118.253
uid = nobody
gid = nobody
use chroot = yes
read only = yes
max connections=10
[web1]
comment=Web content
path=/var/www/001
ignore errors
auth users=tom,jerry
secrets file=/etc/rsyncd.secrets
hosts allow=192.168.118.254
hosts deny=*
list=false
[root@localhost /]# echo "tom:123">/etc/rsyncd.secrets
[root@localhost /]# echo "jerry:123">/etc/rsyncd.secrets
为了安全最好设置为600
[root@localhost /]# chmod 600 /etc/rsyncd.secrets
[root@localhost /]# echo "welcome to access">/etc/rsyncd.motd
[root@localhost /]# rsync --daemon
[root@localhost /]# echo "/usr/bin/rsync --daemon">/etc/rc.local
[root@localhost /]# iptables -I INPUT -p tcp --dport 873 -j ACCEPT
[root@localhost /]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
在Web服务器192.168.118.250中配置如下:
[root@test ~]# mkdir -p /var/www/002
[root@test ~]# chmod 660 /var/www/002
[root@test ~]# chown nobody.nobody /var/www/002
[root@test ~]# gedit /etc/rsyncd.conf
修改内容如下:
#/etc/rsyncd.conf
#
transfer logging=yes
log file=/var/log/rsyncd.log
pid file=/var/run/rsyncd.pid
lock file=/var/run/rsync.lock
port=873
address = 192.168.118.250
uid = nobody
gid = nobody
use chroot = yes
read only = yes
max connections=10
[web1]
comment=Web content
path=/var/www/002
ignore errors
auth users=tom,jerry
secrets file=/etc/rsyncd.secrets
hosts allow=192.168.118.254
hosts deny=*
list=false
[root@test ~]# echo "tom:123">/etc/rsyncd.secrets
[root@test ~]# echo "jerry:123">/etc/rsyncd.secrets
[root@test ~]# chmod 600 /etc/rsyncd.secrets
[root@test ~]# echo "welcome to access">/etc/rsyncd.motd
[root@test ~]# rsync --daemon
[root@test ~]# echo "/usr/bin/rsync --daemon">/etc/rc.local
[root@test ~]# iptables -I INPUT -p tcp --dport 873 -j ACCEPT
[root@test ~]# service iptables save
iptables:将防火墙规则保存到 /etc/sysconfig/iptables: [确定]
[root@test ~]#
在192.168.118.254上面配置
root@ubuntu:/home/tempal/inotify-tools-master# apt-get install automake libtool
root@ubuntu:/home/tempal/inotify-tools-master# sh autogen.sh
root@ubuntu:/home/tempal/inotify-tools-master# ./configure
root@ubuntu:/home/tempal/inotify-tools-master# make &&make install
root@ubuntu:/home/tempal/inotify-tools-master# echo "123">/root/rsync.pass
root@ubuntu:/home/tempal/inotify-tools-master# cat /root/rsync.pass
123
root@ubuntu:/home/tempal/inotify-tools-master# chmod 600 /root/rsync.pass
root@ubuntu:/home/tempal/inotify-tools-master# cd ..
root@ubuntu:/home/tempal# gedit notify_rsync.sh
添加内容:
#!/bin/bash
#this rsync scritpt based on inotify
#Data:2014-6-27
export PATH=/bin:/usr/bin:/usr/local/bin
SRC=http://www.mamicode.com/web_data/
DEST1=web1
DEST2=web2
Client1=192.168.118.253
Client2=192.168.118.250
User=jerry
#password file must not be other-accessible
Passfile=/root/rsync.pass
#if the DEST direct not found,then create one.
[ ! -e $Passfile ] && exit 2
#wait for change
inotifywait -mrq --timefmt ‘%y-%m-%d %H:%M‘ --format ‘%T %w%f %e‘ \
--event modify,create,move,delete,attrib $SRC|while read line
do
echo "$line">/var/log/inotify_web 2>&1
/usr/bin/rsync -avz --delete --progress --password-file=$Passfile $SRC${User}@$Client1::$DEST1 >>/var/log/sync_web1 2>&1
/usr/bin/rsync -avz --delete --progress --password-file=$Passfile $SRC${User}@$Client2::$DEST2 >>/var/log/sync_web2 2>&1
done &
执行文本,并开机启动
root@ubuntu:/home/tempal# sh notify_rsync.sh
root@ubuntu:/home/tempal# echo "/root/notify_rsync.sh" /etc/rc.local
如果出现这个错误“/usr/local/bin/inotifywait: error while loading shared libraries: libinotifytools.so.0”可以采用以下办法解决:
ln -sv /usr/local/lib/libinotify* /usr/lib/
ln -s /usr/local/lib/libinotifytools.so.0 /usr/lib/libinotifytools.so.0
cp /usr/lib/libinotifytools.so.0 /usr/local/lib/