最近做日志分析，发现logstash较符合自己的需求，

•  Logstash：做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助.一般使用shipper作为log收集、indexer作为log转载.  

• Logstash shipper收集log 并将log转发给redis 存储  

• Logstash indexer从redis中读取数据并转发给elasticsearch  

• redis：是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。  

• Elasticsearch：elasticsearch是基于lucene的开源搜索引擎,用来做索引。

• Kibana： 开源web展现,界面很漂亮,是一个功能强大的elasticsearch数据显示客户端，logstash已经内置了kibana，你也可以单独部署kibana，最新版的kibana3是纯html+js客户端.

软件下载目录

http://www.elasticsearch.org/downloads/

我的环境如下

os：centos6.3_x86-64

redis-2.8.7.tar.gz    

kibana-3.0.0

java version "1.7.0_51"

elasticsearch-0.90.12

一，安装java

yum -y install java

二,安装redis

cd ~/src
wget http://download.redis.io/releases/redis-2.8.7.tar.gz
tar -zxf redis-2.8.7.tar.gz
cd redis-2.8.7.tar.gz
make
sudo make install

安装完毕后

/etc/init.d/redis_6379 start

测试是否正常

[root@file1 ~]# redis-cli ping
PONG
[root@file1 ~]#

[root@file1 ~]# netstat -tanpu|grep redis
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      1391/redis-server *

三，安装Elasticsearch

cd /search
sudo mkdir elasticsearch
cd elasticsearch
sudo wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.12.zip
sudo unzip elasticsearch-0.90.12.zip

备注：当开始使用的是1.x.x java报错，后来用的0.9.。

https://groups.google.com/forum/#!topic/logstash-users/fvFT7pgQTEM

Are you using elasticsearch_http for your output?  If not, 1.3.3 is based on 0.90.x elasticsearch, and won’t play nice with 1.0.x elasticsearch with just the “elasticsearch” output.

启动ES服务器

切换到elasticsearch目录运行

bin/elasticsearch -f

默认端口是9200

curl -X GET http://localhost:9200
[root@file1 ~]# curl -X GET http://localhost:9200
{
  "ok" : true,
  "status" : 200,
  "name" : "Master Pandemonium",
  "version" : {
    "number" : "0.90.12",
    "build_hash" : "26feed79983063ae83bfa11bd4ce214b1f45c884",
    "build_timestamp" : "2014-02-25T15:38:23Z",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

四.安装logstash

cd /search
sudo mkdir logstash
cd logstash
sudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar

新建配置文件index.conf

# This is the logstash server index configuration.
# This file will be put in the same folder with logtash.jar file in the
# /etc/logtash/
# This takes information straight from redis and loads it into elasticsearch.
input {
  redis {
    host => "127.0.0.1"
    type => "syslog"
    threads => 4
    # these settings should match the output of the agent
    data_type => "list"
    key => "logstash"
    # We use json_event here since the sender is a logstash agent
    format => "json_event"
    }
}
output {
  elasticsearch {
    host => "127.0.0.1"
  }
}

新建shiper.conf

input {
  stdin {
    type => "test"
  }
}
output {
  stdout { codec => rubydebug }
  redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}

运行配置

java -jar logstash.jar agent -f shipper.conf
java -jar logstash.jar agent -f index.conf

五，配置kibana

logstash的最新版已经内置kibana，你也可以单独部署kibana。kibana3是纯粹JavaScript+html的客户端，所以可以部署到任意http服务器上。

https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.zip

解压到web目录

http://127.0.0.1/kibana/index.html

资料来源参考：

1.http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html

2.http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

3.http://my.oschina.net/guol/blog/179848

4.http://tinytub.github.io/logstash-install.html

5.install in Ubuntu server

http://tips4admin.com/blog/2013/10/how-to-centralize-your-log-with-logstash-elasticsearch-redis-kibana-in-ubuntu-server/  

6.logstash官方文档 the logstash book

本文出自 “静待花开” 博客，请务必保留此出处http://motor.blog.51cto.com/729413/1411199