首页 > 代码库 > 使用Nginx和Logstash以及kafka来实现网站日志采集的详细步骤和过程
使用Nginx和Logstash以及kafka来实现网站日志采集的详细步骤和过程
使用Nginx和Logstash以及kafka来实现网站日志采集的详细步骤和过程 环境介绍： linux虚拟机3台，主机名分别为hadoop01、hadoop02和hadoop03; 在这3台虚拟机上分别部署了3个Zookeeper，这里Zookeeper的具体安装步骤不做介绍; 在这3台虚拟机上分别部署了3个kafka，这里kafka的具体安装步骤也不做介绍; 我们在hadoop02这台机器上安装一个Logstash，其安装过程非常简单，解压既可使用; ======================================================================================================== 首先是在hadoop02这台机器上部署一个nginx，不过这个nginx不是一般的nginx，它使用了lua等插件。 所以其安装步骤比较复杂，在本文的后面我们会有详细的安装步骤和安装记录。 假如现在我们hadoop02这台机器上的nginx安装好了，我们来修改nginx的配置文件"nginx.conf"。 将"nginx.conf"中原来的内容都清除，用下面的内容替换： worker_processes 2; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; log_format tick "$msec^A$remote_addr^A$u_domain^A$u_url^A$u_title^A$u_referrer^A$u_sh^A$u_sw^A$u_cd^A$u_lang^A$http_user_agent^A$u_utrace^A$u_account"; access_log logs/access.log tick; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location /log.gif { default_type image/gif; access_log off; access_by_lua " local uid = ngx.var.cookie___utrace if not uid then uid = ngx.md5(ngx.now() .. ngx.var.remote_addr .. ngx.var.http_user_agent) end ngx.header[‘Set-Cookie‘] = {‘__utrace=‘ .. uid .. ‘; path=/‘} if ngx.var.arg_domain then ngx.location.capture(‘/i-log?‘ .. ngx.var.args .. ‘&utrace=‘ .. uid) end "; add_header Expires "Fri, 01 Jan 1980 00:00:00 GMT"; add_header Pragma "no-cache"; add_header Cache-Control "no-cache, max-age=0, must-revalidate"; empty_gif; } location /i-log { internal; set_unescape_uri $u_domain $arg_domain; set_unescape_uri $u_url $arg_url; set_unescape_uri $u_title $arg_title; set_unescape_uri $u_referrer $arg_referrer; set_unescape_uri $u_sh $arg_sh; set_unescape_uri $u_sw $arg_sw; set_unescape_uri $u_cd $arg_cd; set_unescape_uri $u_lang $arg_lang; set_unescape_uri $u_utrace $arg_utrace; set_unescape_uri $u_account $arg_account; log_subrequest on; access_log /var/nginx_logs/track.log tick; echo ‘‘; } } } 在hadoop02机器的"/var/nginx_logs/"目录下面创建一个"track.log"文件。 在hadoop02机器的"/usr/local/nginx/html/"目录下面新建一个"track.js"文件，文件内容如下： (function () { var params = {}; //Document对象数据 if(document) { params.domain = document.domain || ‘‘; params.url = document.URL || ‘‘; params.title = document.title || ‘‘; params.referrer = document.referrer || ‘‘; } //Window对象数据 if(window && window.screen) { params.sh = window.screen.height || 0; params.sw = window.screen.width || 0; params.cd = window.screen.colorDepth || 0; } //navigator对象数据 if(navigator) { params.lang = navigator.language || ‘‘; } //解析_maq配置 if(_maq) { for(var i in _maq) { switch(_maq[i][0]) { case ‘_setAccount‘: params.account = _maq[i][1]; break; default: break; } } } //拼接参数串 var args = ‘‘; for(var i in params) { if(args != ‘‘) { args += ‘&‘; } args += i + ‘=‘ + encodeURIComponent(params[i]); } //通过Image对象请求后端脚本 var img = new Image(1, 1); img.src = ‘http://hadoop02/log.gif?‘ + args; })(); [root@hadoop02 conf]# [root@hadoop02 conf]# cd ../html/ [root@hadoop02 html]# ll total 12 -rw-r--r--. 1 root root 537 Dec 31 08:02 50x.html -rw-r--r--. 1 root root 612 Dec 31 08:02 index.html -rw-r--r--. 1 root root 1237 Dec 31 08:16 track.js 在要统计的页面添加如下js脚本，假如我们在"taotao-portal"项目的"index.jsp"的最下面的"</body>"标签前面添加该脚本： <script type="text/javascript"> var _maq = _maq || []; _maq.push([‘_setAccount‘, ‘zx5352‘]); (function() { var ma = document.createElement(‘script‘); ma.type = ‘text/javascript‘; ma.async = true; ma.src = ‘http://hadoop02/track.js‘; var s = document.getElementsByTagName(‘script‘)[0]; s.parentNode.insertBefore(ma, s); })(); </script> 在3台机器上的/root/apps/目录下面分别部署一个kafka，并使用如下命令启动3台机器上的kafka服务器： /root/apps/kafka/bin/kafka-server-start.sh /root/apps/kafka/config/server.properties 在任意一台kafka服务器上查看一下kafka中的topic： /root/apps/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181 在任意一台kafka服务器上创建一个名为accesslog的topic： /root/apps/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic accesslog 在任意一台kafka服务器上再查看一下kafka中的topic是否创建成功： /root/apps/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181 使用如下命令启动hadoop02机器上的nginx服务器： /usr/local/nginx/sbin/nginx 然后在浏览器中访问如下地址，应该可以看到nginx的主界面： http://hadoop02/ 然后我们再在浏览器中访问如下地址，应该可以看到track.js中的内容： http://hadoop02/track.js 将Logstash部署到hadoop02的/root/apps/目录下; 在hadoop02机器的/root/apps/目录下创建flow-kafka.conf文件，文件内容如下： input { file { path => "/var/nginx_logs/*.log" discover_interval => 5 start_position => "beginning" } } output { kafka { topic_id => "accesslog" codec => plain { format => "%{message}" } bootstrap_servers => "192.168.71.11:9092,192.168.71.12:9092,192.168.71.13:9092" } } 启动Logstash，并指定刚才创建的flow-kafka.conf文件： logstash/bin/logstash -f flow-kafka.conf 在任意一台机器上启动一个kafka的消费者，来消费"accesslog"这个topic中的消息： /root/apps/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic accesslog --from-beginning 将我们的"taotao-portal"项目所在tomcat服务器启动，访问首页的index.jsp页面， 此时在"hadoop02"的"/var/nginx_logs/"目录下面应该看到"track.log"文件会有日志输出。 并且在我们上面启动的那个kafka的消费者会有消费被消费。 我们的"taotao-portal"项目位于hadoop01这台服务器上，所以kafka消费者消费到如下消息： 1483210782.201^A192.168.71.1^Ahadoop01^Ahttp://hadoop01:8808/^A\xE6\xB7\x98\xE6\xB7\x98\xE7\xBD\x91\xE4\xB8\x8A\xE5\x95\x86\xE5\x9F\x8E-\xE7\xBB\xBC\xE5\x90\x88\xE7\xBD\x91\xE8\xB4\xAD\xE9\xA6\x96\xE9\x80\x89\xEF\xBC\x88JD.COM\xEF\xBC\x89-\xE6\xAD\xA3\xE5\x93\x81\xE4\xBD\x8E\xE4\xBB\xB7\xE3\x80\x81\xE5\x93\x81\xE8\xB4\xA8\xE4\xBF\x9D\xE9\x9A\x9C\xE3\x80\x81\xE8\xB4\xA7\xE5\x88\xB0\xE4\xBB\x98\xE6\xAC\xBE\xE3\x80\x81\xE9\x85\x8D\xE9\x80\x81\xE5\x8F\x8A\xE6\x97\xB6\xE3\x80\x81\xE6\x94\xBE\xE5\xBF\x83\xE6\x9C\x8D\xE5\x8A\xA1\xE3\x80\x81\xE8\xBD\xBB\xE6\x9D\xBE\xE8\xB4\xAD\xE7\x89\xA9\xEF\xBC\x81^A^A768^A1366^A24^Azh-CN^AMozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0^A4af03b636b4f947d47111ad4c15a31b5^Azx5352 我们将上面的消息通过"^A"进行分割，可以得到如下字段： 1483210782.201 192.168.71.1 hadoop01 http://hadoop01:8808/ \xE6\xB7\x98\xE6\xB7\x98\xE7\xBD\x91\xE4\xB8\x8A\xE5\x95\x86\xE5\x9F\x8E-\xE7\xBB\xBC\xE5\x90\x88\xE7\xBD\x91\xE8\xB4\xAD\xE9\xA6\x96\xE9\x80\x89\xEF\xBC\x88JD.COM\xEF\xBC\x89-\xE6\xAD\xA3\xE5\x93\x81\xE4\xBD\x8E\xE4\xBB\xB7\xE3\x80\x81\xE5\x93\x81\xE8\xB4\xA8\xE4\xBF\x9D\xE9\x9A\x9C\xE3\x80\x81\xE8\xB4\xA7\xE5\x88\xB0\xE4\xBB\x98\xE6\xAC\xBE\xE3\x80\x81\xE9\x85\x8D\xE9\x80\x81\xE5\x8F\x8A\xE6\x97\xB6\xE3\x80\x81\xE6\x94\xBE\xE5\xBF\x83\xE6\x9C\x8D\xE5\x8A\xA1\xE3\x80\x81\xE8\xBD\xBB\xE6\x9D\xBE\xE8\xB4\xAD\xE7\x89\xA9\xEF\xBC\x81 768 1366 24 zh-CN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0 4af03b636b4f947d47111ad4c15a31b5 zx5352 ======================================================================================================== 下面是nginx服务器的详细安装记录： Last login: Sat Dec 31 06:26:15 2016 from 192.168.71.1 由于我们要使用本地yum仓库来安装，所有首先看我们的yum源的配置： [root@hadoop02 ~]# cd /etc/yum.repos.d/ [root@hadoop02 yum.repos.d]# ll total 24 -rw-r--r--. 1 root root 1991 Aug 3 2015 CentOS-Base.repo -rw-r--r--. 1 root root 647 Aug 3 2015 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 289 Aug 3 2015 CentOS-fasttrack.repo -rw-r--r--. 1 root root 630 Aug 3 2015 CentOS-Media.repo -rw-r--r--. 1 root root 6259 Aug 3 2015 CentOS-Vault.repo 将所有的".repo"文件都改名，只保留一个"CentOS-Media.repo"文件： [root@hadoop02 yum.repos.d]# mv CentOS-Base.repo CentOS-Base.repo.bak [root@hadoop02 yum.repos.d]# mv CentOS-Debuginfo.repo CentOS-Debuginfo.repo.bak [root@hadoop02 yum.repos.d]# mv CentOS-fasttrack.repo CentOS-fasttrack.repo.bak [root@hadoop02 yum.repos.d]# mv CentOS-Vault.repo CentOS-Vault.repo.bak [root@hadoop02 yum.repos.d]# ll total 24 -rw-r--r--. 1 root root 1991 Aug 3 2015 CentOS-Base.repo.bak -rw-r--r--. 1 root root 647 Aug 3 2015 CentOS-Debuginfo.repo.bak -rw-r--r--. 1 root root 289 Aug 3 2015 CentOS-fasttrack.repo.bak -rw-r--r--. 1 root root 630 Aug 3 2015 CentOS-Media.repo -rw-r--r--. 1 root root 6259 Aug 3 2015 CentOS-Vault.repo.bak 查看我们的"CentOS-Media.repo"文件的内容： [root@hadoop02 yum.repos.d]# vi CentOS-Media.repo [c6-media] name=CentOS-$releasever - Media baseurl=file:///media/CentOS/ file:///media/cdrom/ file:///media/cdrecorder/ gpgcheck=1 enabled=0 用下面的内容替换CentOS-Media.repo文件原有的内容，指定我们的yum仓库位于/mnt/cdrom/下： [c6-media] name=CentOS-$releasever - Media baseurl=file:///mnt/cdrom/ gpgcheck=1 enabled=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 ~ ~ ~ ~ "CentOS-Media.repo" 19L, 564C written 在虚拟机上设置我们的iso光盘镜像文件，并勾选"启动时连接"复选框，重启虚拟机。 [root@hadoop02 yum.repos.d]# cd /mnt [root@hadoop02 mnt]# ll total 0 在/mnt/下面创建cdrom目录： [root@hadoop02 mnt]# mkdir cdrom [root@hadoop02 mnt]# ll total 4 drwxr-xr-x. 2 root root 4096 Dec 31 07:45 cdrom [root@hadoop02 mnt]# pwd /mnt 将光盘镜像挂载到/mnt/cdrom目录： [root@hadoop02 mnt]# mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom [root@hadoop02 mnt]# cd cdrom/ 查看/mnt/cdrom目录下是否能看到光盘镜像中的内容，如果能看到，说明挂载成功。 [root@hadoop02 cdrom]# ll total 558 -r--r--r--. 2 root root 14 Aug 4 2015 CentOS_BuildTag dr-xr-xr-x. 3 root root 2048 Aug 4 2015 EFI -r--r--r--. 2 root root 212 Nov 27 2013 EULA -r--r--r--. 2 root root 18009 Nov 27 2013 GPL dr-xr-xr-x. 3 root root 2048 Aug 4 2015 images dr-xr-xr-x. 2 root root 2048 Aug 4 2015 isolinux dr-xr-xr-x. 2 root root 528384 Aug 4 2015 Packages -r--r--r--. 2 root root 1354 Jul 25 2015 RELEASE-NOTES-en-US.html dr-xr-xr-x. 2 root root 4096 Aug 4 2015 repodata -r--r--r--. 2 root root 1706 Nov 27 2013 RPM-GPG-KEY-CentOS-6 -r--r--r--. 2 root root 1730 Nov 27 2013 RPM-GPG-KEY-CentOS-Debug-6 -r--r--r--. 2 root root 1730 Nov 27 2013 RPM-GPG-KEY-CentOS-Security-6 -r--r--r--. 2 root root 1734 Nov 27 2013 RPM-GPG-KEY-CentOS-Testing-6 -r--r--r--. 1 root root 3380 Aug 4 2015 TRANS.TBL 下面我们就随便安装一个"telnet"软件，如果能够安装成功，说明我的本地yum仓库配置好了，下面我们就可以来安装我们的nginx了。 [root@hadoop02 cdrom]# yum install telnet Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process Determining fastest mirrors c6-media | 4.0 kB 00:00 ... c6-media/primary_db | 4.6 MB 00:00 ... Resolving Dependencies --> Running transaction check ---> Package telnet.x86_64 1:0.17-48.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Installing: telnet x86_64 1:0.17-48.el6 c6-media 58 k Transaction Summary ======================================================================================================================================================================= Install 1 Package(s) Total download size: 58 k Installed size: 109 k Is this ok [y/N]: y Downloading Packages: warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Importing GPG key 0xC105B9DE: Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org> Package: centos-release-6-7.el6.centos.12.3.x86_64 (@anaconda-CentOS-201508042137.x86_64/6.7) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Warning: RPMDB altered outside of yum. ** Found 3 pre-existing rpmdb problem(s), ‘yum check‘ output follows: 2:postfix-2.6.6-6.el6_5.x86_64 has missing requires of libmysqlclient.so.16()(64bit) 2:postfix-2.6.6-6.el6_5.x86_64 has missing requires of libmysqlclient.so.16(libmysqlclient_16)(64bit) 2:postfix-2.6.6-6.el6_5.x86_64 has missing requires of mysql-libs Installing : 1:telnet-0.17-48.el6.x86_64 1/1 Verifying : 1:telnet-0.17-48.el6.x86_64 1/1 Installed: telnet.x86_64 1:0.17-48.el6 Complete! 下面我们来使用yum命令来安装一些编译环境，这是后面通过编译安装nginx以及其他软件所需要的一些环境。 [root@hadoop02 cdrom]# yum -y install gcc perl pcre-devel openssl openssl-devel Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process Loading mirror speeds from cached hostfile Package gcc-4.4.7-16.el6.x86_64 already installed and latest version Package 4:perl-5.10.1-141.el6.x86_64 already installed and latest version Package openssl-1.0.1e-42.el6.x86_64 already installed and latest version Resolving Dependencies --> Running transaction check ---> Package openssl-devel.x86_64 0:1.0.1e-42.el6 will be installed --> Processing Dependency: zlib-devel for package: openssl-devel-1.0.1e-42.el6.x86_64 --> Processing Dependency: krb5-devel for package: openssl-devel-1.0.1e-42.el6.x86_64 ---> Package pcre-devel.x86_64 0:7.8-7.el6 will be installed --> Running transaction check ---> Package krb5-devel.x86_64 0:1.10.3-42.el6 will be installed --> Processing Dependency: libselinux-devel for package: krb5-devel-1.10.3-42.el6.x86_64 --> Processing Dependency: libcom_err-devel for package: krb5-devel-1.10.3-42.el6.x86_64 --> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.10.3-42.el6.x86_64 ---> Package zlib-devel.x86_64 0:1.2.3-29.el6 will be installed --> Running transaction check ---> Package keyutils-libs-devel.x86_64 0:1.4-5.el6 will be installed ---> Package libcom_err-devel.x86_64 0:1.41.12-22.el6 will be installed ---> Package libselinux-devel.x86_64 0:2.0.94-5.8.el6 will be installed --> Processing Dependency: libsepol-devel >= 2.0.32-1 for package: libselinux-devel-2.0.94-5.8.el6.x86_64 --> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.0.94-5.8.el6.x86_64 --> Running transaction check ---> Package libsepol-devel.x86_64 0:2.0.41-4.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================= Package Arch Version Repository Size ======================================================================================================================================================================= Installing: openssl-devel x86_64 1.0.1e-42.el6 c6-media 1.2 M pcre-devel x86_64 7.8-7.el6 c6-media 320 k Installing for dependencies: keyutils-libs-devel x86_64 1.4-5.el6 c6-media 29 k krb5-devel x86_64 1.10.3-42.el6 c6-media 502 k libcom_err-devel x86_64 1.41.12-22.el6 c6-media 33 k libselinux-devel x86_64 2.0.94-5.8.el6 c6-media 137 k libsepol-devel x86_64 2.0.41-4.el6 c6-media 64 k zlib-devel x86_64 1.2.3-29.el6 c6-media 44 k Transaction Summary ======================================================================================================================================================================= Install 8 Package(s) Total download size: 2.3 M Installed size: 5.9 M Downloading Packages: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 6.0 MB/s | 2.3 MB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : zlib-devel-1.2.3-29.el6.x86_64 1/8 Installing : libcom_err-devel-1.41.12-22.el6.x86_64 2/8 Installing : libsepol-devel-2.0.41-4.el6.x86_64 3/8 Installing : libselinux-devel-2.0.94-5.8.el6.x86_64 4/8 Installing : keyutils-libs-devel-1.4-5.el6.x86_64 5/8 Installing : krb5-devel-1.10.3-42.el6.x86_64 6/8 Installing : openssl-devel-1.0.1e-42.el6.x86_64 7/8 Installing : pcre-devel-7.8-7.el6.x86_64 8/8 Verifying : keyutils-libs-devel-1.4-5.el6.x86_64 1/8 Verifying : libsepol-devel-2.0.41-4.el6.x86_64 2/8 Verifying : libcom_err-devel-1.41.12-22.el6.x86_64 3/8 Verifying : krb5-devel-1.10.3-42.el6.x86_64 4/8 Verifying : openssl-devel-1.0.1e-42.el6.x86_64 5/8 Verifying : libselinux-devel-2.0.94-5.8.el6.x86_64 6/8 Verifying : zlib-devel-1.2.3-29.el6.x86_64 7/8 Verifying : pcre-devel-7.8-7.el6.x86_64 8/8 Installed: openssl-devel.x86_64 0:1.0.1e-42.el6 pcre-devel.x86_64 0:7.8-7.el6 Dependency Installed: keyutils-libs-devel.x86_64 0:1.4-5.el6 krb5-devel.x86_64 0:1.10.3-42.el6 libcom_err-devel.x86_64 0:1.41.12-22.el6 libselinux-devel.x86_64 0:2.0.94-5.8.el6 libsepol-devel.x86_64 0:2.0.41-4.el6 zlib-devel.x86_64 0:1.2.3-29.el6 Complete! 我们把准备好的一些和nginx相关的软件都放在了/root/apps/elk/soft/目录下面： [root@hadoop02 cdrom]# cd /root/apps/elk/ [root@hadoop02 elk]# ll total 4 drwxr-xr-x. 2 root root 4096 Dec 31 07:38 soft [root@hadoop02 elk]# cd soft/ [root@hadoop02 soft]# ll total 5912 -rw-r--r--. 1 root root 64779 Dec 31 07:36 echo-nginx-module-0.58.tar.gz -rw-r--r--. 1 root root 847615 Dec 31 07:36 LuaJIT-2.0.4.tar.gz -rw-r--r--. 1 root root 569372 Dec 31 07:36 lua-nginx-module-0.10.0.tar.gz -rw-r--r--. 1 root root 833473 Dec 31 07:36 nginx-1.8.1.tar.gz -rw-r--r--. 1 root root 2450 Dec 31 07:36 nginx.conf -rw-r--r--. 1 root root 65029 Dec 31 07:36 ngx_devel_kit-0.2.19.tar.gz -rw-r--r--. 1 root root 3616491 Dec 31 07:36 openresty-1.9.7.3.tar.gz -rw-r--r--. 1 root root 35495 Dec 31 07:36 set-misc-nginx-module-0.29.tar.gz -rw-r--r--. 1 root root 7107 Dec 31 07:36 网站统计.txt [root@hadoop02 soft]# [root@hadoop02 soft]# [root@hadoop02 soft]# 开始解压"LuaJIT-2.0.4.tar.gz"安装包： [root@hadoop02 soft]# tar -zxvf LuaJIT-2.0.4.tar.gz -C /usr/local/src/ 解压的过程省略。解压完成后，进入到解压后的"LuaJIT-2.0.4"目录： [root@hadoop02 soft]# cd /usr/local/src/LuaJIT-2.0.4/ 编译"LuaJIT-2.0.4"并将其安装到"/usr/local/luajit"目录下： [root@hadoop02 LuaJIT-2.0.4]# make && make install PREFIX=/usr/local/luajit ==== Building LuaJIT 2.0.4 ==== make -C src make[1]: Entering directory `/usr/local/src/LuaJIT-2.0.4/src‘ HOSTCC host/minilua.o HOSTLINK host/minilua DYNASM host/buildvm_arch.h HOSTCC host/buildvm.o HOSTCC host/buildvm_asm.o HOSTCC host/buildvm_peobj.o HOSTCC host/buildvm_lib.o HOSTCC host/buildvm_fold.o HOSTLINK host/buildvm ..................... ..................... ..................... CC lib_init.o AR libluajit.a CC luajit.o BUILDVM jit/vmdef.lua DYNLINK libluajit.so LINK luajit OK Successfully built LuaJIT make[1]: Leaving directory `/usr/local/src/LuaJIT-2.0.4/src‘ ==== Successfully built LuaJIT 2.0.4 ==== ==== Installing LuaJIT 2.0.4 to /usr/local/luajit ==== mkdir -p /usr/local/luajit/bin /usr/local/luajit/lib /usr/local/luajit/include/luajit-2.0 /usr/local/luajit/share/man/man1 /usr/local/luajit/lib/pkgconfig /usr/local/luajit/share/luajit-2.0.4/jit /usr/local/luajit/share/lua/5.1 /usr/local/luajit/lib/lua/5.1 cd src && install -m 0755 luajit /usr/local/luajit/bin/luajit-2.0.4 cd src && test -f libluajit.a && install -m 0644 libluajit.a /usr/local/luajit/lib/libluajit-5.1.a || : rm -f /usr/local/luajit/bin/luajit /usr/local/luajit/lib/libluajit-5.1.so.2.0.4 /usr/local/luajit/lib/libluajit-5.1.so /usr/local/luajit/lib/libluajit-5.1.so cd src && test -f libluajit.so && install -m 0755 libluajit.so /usr/local/luajit/lib/libluajit-5.1.so.2.0.4 && ldconfig -n /usr/local/luajit/lib && ln -sf libluajit-5.1.so.2.0.4 /usr/local/luajit/lib/libluajit-5.1.so && ln -sf libluajit-5.1.so.2.0.4 /usr/local/luajit/lib/libluajit-5.1.so || : cd etc && install -m 0644 luajit.1 /usr/local/luajit/share/man/man1 cd etc && sed -e "s|^prefix=.*|prefix=/usr/local/luajit|" -e "s|^multilib=.*|multilib=lib|" luajit.pc > luajit.pc.tmp && install -m 0644 luajit.pc.tmp /usr/local/luajit/lib/pkgconfig/luajit.pc && rm -f luajit.pc.tmp cd src && install -m 0644 lua.h lualib.h lauxlib.h luaconf.h lua.hpp luajit.h /usr/local/luajit/include/luajit-2.0 cd src/jit && install -m 0644 bc.lua v.lua dump.lua dis_x86.lua dis_x64.lua dis_arm.lua dis_ppc.lua dis_mips.lua dis_mipsel.lua bcsave.lua vmdef.lua /usr/local/luajit/share/luajit-2.0.4/jit ln -sf luajit-2.0.4 /usr/local/luajit/bin/luajit ==== Successfully installed LuaJIT 2.0.4 to /usr/local/luajit ==== [root@hadoop02 LuaJIT-2.0.4]# [root@hadoop02 local]# cd /root/apps/elk/soft/ [root@hadoop02 soft]# ll total 5912 -rw-r--r--. 1 root root 64779 Dec 31 07:36 echo-nginx-module-0.58.tar.gz -rw-r--r--. 1 root root 847615 Dec 31 07:36 LuaJIT-2.0.4.tar.gz -rw-r--r--. 1 root root 569372 Dec 31 07:36 lua-nginx-module-0.10.0.tar.gz -rw-r--r--. 1 root root 833473 Dec 31 07:36 nginx-1.8.1.tar.gz -rw-r--r--. 1 root root 2450 Dec 31 07:36 nginx.conf -rw-r--r--. 1 root root 65029 Dec 31 07:36 ngx_devel_kit-0.2.19.tar.gz -rw-r--r--. 1 root root 3616491 Dec 31 07:36 openresty-1.9.7.3.tar.gz -rw-r--r--. 1 root root 35495 Dec 31 07:36 set-misc-nginx-module-0.29.tar.gz -rw-r--r--. 1 root root 7107 Dec 31 07:36 网站统计.txt [root@hadoop02 soft]# 将"lua-nginx-module-0.10.0.tar.gz"解压到"/usr/local/nginx/modules/"目录下: [root@hadoop02 soft]# tar -zxvf lua-nginx-module-0.10.0.tar.gz -C /usr/local/nginx/modules/ .................... .................... [root@hadoop02 soft]# tar -zxvf set-misc-nginx-module-0.29.tar.gz -C /usr/local/nginx/modules/ .................... .................... [root@hadoop02 soft]# tar -zxvf ngx_devel_kit-0.2.19.tar.gz -C /usr/local/nginx/modules/ .................... .................... [root@hadoop02 soft]# tar -zxvf echo-nginx-module-0.58.tar.gz -C /usr/local/nginx/modules/ .................... .................... [root@hadoop02 soft]# tar -zxvf openresty-1.9.7.3.tar.gz -C /usr/local/src/ [root@hadoop02 soft]# cd /usr/local/src/openresty-1.9.7.3/ 将"openresty-1.9.7.3"编译安装到"/usr/local/openresty"目录下： [root@hadoop02 openresty-1.9.7.3]# ./configure --prefix=/usr/local/openresty --with-luajit && make && make install ................................ ................................ make[1]: Entering directory `/usr/local/src/openresty-1.9.7.3/build/lua-resty-core-0.1.4‘ /usr/local/src/openresty-1.9.7.3/build/install -d /usr/local/openresty/lualib/resty/core/ /usr/local/src/openresty-1.9.7.3/build/install -d /usr/local/openresty/lualib/ngx/ /usr/local/src/openresty-1.9.7.3/build/install lib/resty/*.lua /usr/local/openresty/lualib/resty/ /usr/local/src/openresty-1.9.7.3/build/install lib/resty/core/*.lua /usr/local/openresty/lualib/resty/core/ /usr/local/src/openresty-1.9.7.3/build/install lib/ngx/*.lua /usr/local/openresty/lualib/ngx/ make[1]: Leaving directory `/usr/local/src/openresty-1.9.7.3/build/lua-resty-core-0.1.4‘ cd /usr/local/src/openresty-1.9.7.3/build/lua-resty-upstream-healthcheck-0.03 && make install DESTDIR= LUA_LIB_DIR=/usr/local/openresty/lualib INSTALL=/usr/local/src/openresty-1.9.7.3/build/install make[1]: Entering directory `/usr/local/src/openresty-1.9.7.3/build/lua-resty-upstream-healthcheck-0.03‘ /usr/local/src/openresty-1.9.7.3/build/install -d /usr/local/openresty/lualib/resty/upstream/ /usr/local/src/openresty-1.9.7.3/build/install lib/resty/upstream/*.lua /usr/local/openresty/lualib/resty/upstream/ make[1]: Leaving directory `/usr/local/src/openresty-1.9.7.3/build/lua-resty-upstream-healthcheck-0.03‘ cd /usr/local/src/openresty-1.9.7.3/build/resty-cli-0.05 && /usr/local/src/openresty-1.9.7.3/build/install resty /usr/local/openresty/bin/ cd /usr/local/src/openresty-1.9.7.3/build/nginx-1.9.7 && make install DESTDIR= make[1]: Entering directory `/usr/local/src/openresty-1.9.7.3/build/nginx-1.9.7‘ make -f objs/Makefile install make[2]: Entering directory `/usr/local/src/openresty-1.9.7.3/build/nginx-1.9.7‘ test -d ‘/usr/local/openresty/nginx‘ || mkdir -p ‘/usr/local/openresty/nginx‘ test -d ‘/usr/local/openresty/nginx/sbin‘ || mkdir -p ‘/usr/local/openresty/nginx/sbin‘ test ! -f ‘/usr/local/openresty/nginx/sbin/nginx‘ || mv ‘/usr/local/openresty/nginx/sbin/nginx‘ ‘/usr/local/openresty/nginx/sbin/nginx.old‘ cp objs/nginx ‘/usr/local/openresty/nginx/sbin/nginx‘ test -d ‘/usr/local/openresty/nginx/conf‘ || mkdir -p ‘/usr/local/openresty/nginx/conf‘ cp conf/koi-win ‘/usr/local/openresty/nginx/conf‘ cp conf/koi-utf ‘/usr/local/openresty/nginx/conf‘ cp conf/win-utf ‘/usr/local/openresty/nginx/conf‘ test -f ‘/usr/local/openresty/nginx/conf/mime.types‘ || cp conf/mime.types ‘/usr/local/openresty/nginx/conf‘ cp conf/mime.types ‘/usr/local/openresty/nginx/conf/mime.types.default‘ test -f ‘/usr/local/openresty/nginx/conf/fastcgi_params‘ || cp conf/fastcgi_params ‘/usr/local/openresty/nginx/conf‘ cp conf/fastcgi_params ‘/usr/local/openresty/nginx/conf/fastcgi_params.default‘ test -f ‘/usr/local/openresty/nginx/conf/fastcgi.conf‘ || cp conf/fastcgi.conf ‘/usr/local/openresty/nginx/conf‘ cp conf/fastcgi.conf ‘/usr/local/openresty/nginx/conf/fastcgi.conf.default‘ test -f ‘/usr/local/openresty/nginx/conf/uwsgi_params‘ || cp conf/uwsgi_params ‘/usr/local/openresty/nginx/conf‘ cp conf/uwsgi_params ‘/usr/local/openresty/nginx/conf/uwsgi_params.default‘ test -f ‘/usr/local/openresty/nginx/conf/scgi_params‘ || cp conf/scgi_params ‘/usr/local/openresty/nginx/conf‘ cp conf/scgi_params ‘/usr/local/openresty/nginx/conf/scgi_params.default‘ test -f ‘/usr/local/openresty/nginx/conf/nginx.conf‘ || cp conf/nginx.conf ‘/usr/local/openresty/nginx/conf/nginx.conf‘ cp conf/nginx.conf ‘/usr/local/openresty/nginx/conf/nginx.conf.default‘ test -d ‘/usr/local/openresty/nginx/logs‘ || mkdir -p ‘/usr/local/openresty/nginx/logs‘ test -d ‘/usr/local/openresty/nginx/logs‘ || mkdir -p ‘/usr/local/openresty/nginx/logs‘ test -d ‘/usr/local/openresty/nginx/html‘ || cp -R docs/html ‘/usr/local/openresty/nginx‘ test -d ‘/usr/local/openresty/nginx/logs‘ || mkdir -p ‘/usr/local/openresty/nginx/logs‘ make[2]: Leaving directory `/usr/local/src/openresty-1.9.7.3/build/nginx-1.9.7‘ make[1]: Leaving directory `/usr/local/src/openresty-1.9.7.3/build/nginx-1.9.7‘ [root@hadoop02 openresty-1.9.7.3]# [root@hadoop02 local]# cd /root/apps/elk/soft/ [root@hadoop02 soft]# ll total 5912 -rw-r--r--. 1 root root 64779 Dec 31 07:36 echo-nginx-module-0.58.tar.gz -rw-r--r--. 1 root root 847615 Dec 31 07:36 LuaJIT-2.0.4.tar.gz -rw-r--r--. 1 root root 569372 Dec 31 07:36 lua-nginx-module-0.10.0.tar.gz -rw-r--r--. 1 root root 833473 Dec 31 07:36 nginx-1.8.1.tar.gz -rw-r--r--. 1 root root 2450 Dec 31 07:36 nginx.conf -rw-r--r--. 1 root root 65029 Dec 31 07:36 ngx_devel_kit-0.2.19.tar.gz -rw-r--r--. 1 root root 3616491 Dec 31 07:36 openresty-1.9.7.3.tar.gz -rw-r--r--. 1 root root 35495 Dec 31 07:36 set-misc-nginx-module-0.29.tar.gz -rw-r--r--. 1 root root 7107 Dec 31 07:36 网站统计.txt [root@hadoop02 soft]# [root@hadoop02 soft]# tar -zxvf nginx-1.8.1.tar.gz -C /usr/local/src/ .................... .................... [root@hadoop02 soft]# cd /usr/local/src/nginx-1.8.1/ 将解压后的"nginx-1.8.1"源码编译到"/usr/local/nginx"目录下： [root@hadoop02 nginx-1.8.1]#./configure --prefix=/usr/local/nginx \ --with-ld-opt="-Wl,-rpath,/usr/local/luajit/lib" --add-module=/usr/local/nginx/modules/ngx_devel_kit-0.2.19 --add-module=/usr/local/nginx/modules/lua-nginx-module-0.10.0 --add-module=/usr/local/nginx/modules/set-misc-nginx-module-0.29 --add-module=/usr/local/nginx/modules/echo-nginx-module-0.58 make -j2 make install 执行上面的这一大段命令后，会输入很多的日志信息，以下只截取部分日志信息： ................................ ................................ ................................ objs/addon/src/ngx_http_echo_var.o objs/addon/src/ngx_http_echo_handler.o objs/addon/src/ngx_http_echo_filter.o objs/addon/src/ngx_http_echo_sleep.o objs/addon/src/ngx_http_echo_location.o objs/addon/src/ngx_http_echo_echo.o objs/addon/src/ngx_http_echo_request_info.o objs/addon/src/ngx_http_echo_subrequest.o objs/addon/src/ngx_http_echo_foreach.o objs/ngx_modules.o -L/usr/local/luajit/lib -Wl,-rpath,/usr/local/luajit/lib -Wl,-E -lpthread -lcrypt -L/usr/local/luajit/lib -lluajit-5.1 -lm -ldl -lpcre -lcrypto -lcrypto -lz make[1]: Leaving directory `/usr/local/src/nginx-1.8.1‘ make -f objs/Makefile manpage make[1]: Entering directory `/usr/local/src/nginx-1.8.1‘ sed -e "s|%%PREFIX%%|/usr/local/nginx|" -e "s|%%PID_PATH%%|/usr/local/nginx/logs/nginx.pid|" -e "s|%%CONF_PATH%%|/usr/local/nginx/conf/nginx.conf|" -e "s|%%ERROR_LOG_PATH%%|/usr/local/nginx/logs/error.log|" < man/nginx.8 > objs/nginx.8 make[1]: Leaving directory `/usr/local/src/nginx-1.8.1‘ 开始使用"make install"命令来安装nginx： [root@hadoop02 nginx-1.8.1]# make install make -f objs/Makefile install make[1]: Entering directory `/usr/local/src/nginx-1.8.1‘ test -d ‘/usr/local/nginx‘ || mkdir -p ‘/usr/local/nginx‘ test -d ‘/usr/local/nginx/sbin‘ || mkdir -p ‘/usr/local/nginx/sbin‘ test ! -f ‘/usr/local/nginx/sbin/nginx‘ || mv ‘/usr/local/nginx/sbin/nginx‘ ‘/usr/local/nginx/sbin/nginx.old‘ cp objs/nginx ‘/usr/local/nginx/sbin/nginx‘ test -d ‘/usr/local/nginx/conf‘ || mkdir -p ‘/usr/local/nginx/conf‘ cp conf/koi-win ‘/usr/local/nginx/conf‘ cp conf/koi-utf ‘/usr/local/nginx/conf‘ cp conf/win-utf ‘/usr/local/nginx/conf‘ test -f ‘/usr/local/nginx/conf/mime.types‘ || cp conf/mime.types ‘/usr/local/nginx/conf‘ cp conf/mime.types ‘/usr/local/nginx/conf/mime.types.default‘ test -f ‘/usr/local/nginx/conf/fastcgi_params‘ || cp conf/fastcgi_params ‘/usr/local/nginx/conf‘ cp conf/fastcgi_params ‘/usr/local/nginx/conf/fastcgi_params.default‘ test -f ‘/usr/local/nginx/conf/fastcgi.conf‘ || cp conf/fastcgi.conf ‘/usr/local/nginx/conf‘ cp conf/fastcgi.conf ‘/usr/local/nginx/conf/fastcgi.conf.default‘ test -f ‘/usr/local/nginx/conf/uwsgi_params‘ || cp conf/uwsgi_params ‘/usr/local/nginx/conf‘ cp conf/uwsgi_params ‘/usr/local/nginx/conf/uwsgi_params.default‘ test -f ‘/usr/local/nginx/conf/scgi_params‘ || cp conf/scgi_params ‘/usr/local/nginx/conf‘ cp conf/scgi_params ‘/usr/local/nginx/conf/scgi_params.default‘ test -f ‘/usr/local/nginx/conf/nginx.conf‘ || cp conf/nginx.conf ‘/usr/local/nginx/conf/nginx.conf‘ cp conf/nginx.conf ‘/usr/local/nginx/conf/nginx.conf.default‘ test -d ‘/usr/local/nginx/logs‘ || mkdir -p ‘/usr/local/nginx/logs‘ test -d ‘/usr/local/nginx/logs‘ || mkdir -p ‘/usr/local/nginx/logs‘ test -d ‘/usr/local/nginx/html‘ || cp -R html ‘/usr/local/nginx‘ test -d ‘/usr/local/nginx/logs‘ || mkdir -p ‘/usr/local/nginx/logs‘ make[1]: Leaving directory `/usr/local/src/nginx-1.8.1‘ 我们的nginx终于安装成功了，下面我们进入到nginx的安装目录"/usr/local/nginx/"下面： [root@hadoop02 nginx-1.8.1]# [root@hadoop02 nginx-1.8.1]# cd /usr/local/nginx/ [root@hadoop02 nginx]# ll total 20 drwxr-xr-x. 2 root root 4096 Dec 31 08:02 conf drwxr-xr-x. 2 root root 4096 Dec 31 08:02 html drwxr-xr-x. 2 root root 4096 Dec 31 08:02 logs drwxr-xr-x. 6 root root 4096 Dec 31 07:54 modules drwxr-xr-x. 2 root root 4096 Dec 31 08:02 sbin [root@hadoop02 nginx]# cd conf/ [root@hadoop02 conf]# ll total 60 -rw-r--r--. 1 root root 1034 Dec 31 08:02 fastcgi.conf -rw-r--r--. 1 root root 1034 Dec 31 08:02 fastcgi.conf.default -rw-r--r--. 1 root root 964 Dec 31 08:02 fastcgi_params -rw-r--r--. 1 root root 964 Dec 31 08:02 fastcgi_params.default -rw-r--r--. 1 root root 2837 Dec 31 08:02 koi-utf -rw-r--r--. 1 root root 2223 Dec 31 08:02 koi-win -rw-r--r--. 1 root root 3957 Dec 31 08:02 mime.types -rw-r--r--. 1 root root 3957 Dec 31 08:02 mime.types.default -rw-r--r--. 1 root root 2656 Dec 31 08:02 nginx.conf -rw-r--r--. 1 root root 2656 Dec 31 08:02 nginx.conf.default -rw-r--r--. 1 root root 596 Dec 31 08:02 scgi_params -rw-r--r--. 1 root root 596 Dec 31 08:02 scgi_params.default -rw-r--r--. 1 root root 623 Dec 31 08:02 uwsgi_params -rw-r--r--. 1 root root 623 Dec 31 08:02 uwsgi_params.default -rw-r--r--. 1 root root 3610 Dec 31 08:02 win-utf [root@hadoop02 conf]# cp nginx.conf nginx.conf.bak [root@hadoop02 conf]# vi nginx.conf 下面我们就可以来编辑nginx的配置文件了，配置文件的具体内容请回到本文的开始部分查看。
使用Nginx和Logstash以及kafka来实现网站日志采集的详细步骤和过程
声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉： 投诉/举报 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。