首页 > 代码库 > ELK配置总结

ELK配置总结

  在经过了近半个月的ELK环境的搭建配置,我做了我个人的工作总结,和大家分享一下。


一、命令总结


1.1Es服务端口查看

# netstat -nlpt | grep -E"9200|9300"


1.2Es插件安装和移除

# ./bin/plugin install file:///home/apps/license-2.3.3.zip

# ./bin/plugin install file:///home/apps/marvel-agent-2.3.3.zip


es移除插件

/usr/local/elasticsearch # ./bin/pluginremove plugins/marvel-agent

/usr/local/elasticsearch # ./bin/pluginremove plugins/license



1.3Kibana插件安装和移除

/usr/local/kibana # ./bin/kibana plugin--install marvel --url file:///home/apps/marvel-2.3.3.tar.gz

 

移除插件

./bin/kibana plugin --remove marvel


1.4logstash检查文件是否有错误

/usr/local/logstash/etc # ../bin/logstash-f logstash.conf  --configtest --verbose

(显示“Configuration OK”说明没有问题)


1.5、启动logstash

/usr/local/logstash/etc # ../bin/logstash-f logstash.conf


1.6、验证es服务

#curl -XGET ES-NODE-IP:9200


1.7、查看es集群状态

# curl -XGET ES-NODE-IP:9200/_cluster/health?pretty=true


1.8、添加和删除es索引,nginx-logs为索引名

curl -XPUT http://ES-NODE-IP:9200/nginx-logs/

 

删除es索引

curl -XDELETE http://ES-NODE-IP:9200/nginx-logs/


1.9验证head插件

http://ES-NODE-IP:9200/_plugin/head/

 

1.10、创建elasticsearch账户

# groupadd esuser

# useradd -d /home/esuser -m esuser

# passwd esuser


1.11查看kafka服务启动状态

# jps

9536 Main

15200 Jps

14647 Kafka

8760 Elasticsearch

21177 -- process information unavailable

14316 QuorumPeerMain

5791 QuorumPeerMain


112、查看kafka端口状态

# netstat -nlpt | grep -E"2181|3888"

tcp       0      0 :::2181                 :::*              LISTEN      14316/java         

tcp       0      0 192.168.1.105:3888     :::*              LISTEN      5791/java


1.13、创建、删除、列出主题(topic

创建主题(topic

/usr/local/kafka # bin/kafka-topics.sh--create --zookeeper localhost:2181 --replication-factor 1 --partitions 1--topic test

Created topic "test".


删除主题(topic

/usr/local/kafka # ./bin/kafka-topics.sh--delete --zookeeper 182.180.50.211:2181 --topic nginx-messages

 

列出所有的topic

/usr/local/kafka # bin/kafka-topics.sh--list --zookeeper localhost:2181

Test

 

显示topic的详细信息

/usr/local/kafka # bin/kafka-topics.sh --describe--zookeeper localhost:218


1.14kafka创建生产者和消费者

创建生产者

/usr/local/kafka #bin/kafka-console-producer.sh --broker-list localhost:9092  --topic test

This is a message

 

创建消费者

/usr/local/kafka #  bin/kafka-console-consumer.sh --zookeeperlocalhost:2181 --topic test --from-beginning

This is a message


如果向上边的可以收到来自生产者的消息,就说明基于kafkazookeeper环境配置好了。


1.15、测试日志文件的数据传输

# cp /var/log/messages /home

# >/var/log/messages

# echo "hello kibana aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa">> /var/log/messages


1.16、重新加载nginx配置文件

# ./nginx -s reload


二、  配置汇总

2.1Java环境配置

# vi /etc/profile

export PATH=$PATH:/soft_ins/mysql/bin

JAVA_HOME=/usr/local/jdk1.8.0_101

PATH=$JAVA_HOME/bin:$PATH

CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar

export PATH JAVA_HOME CLASSPATH

 

# source /etc/profile

 

# java -version

java version "1.8.0_101"

Java(TM) SE Runtime Environment (build1.8.0_101-b13)

Java HotSpot(TM) 64-Bit Server VM (build25.101-b13, mixed mode)


2.2es配置

# vi config/elasticsearch.yml

cluster.name: es_cluster

node.name: node3

path.data: /usr/local/elasticsearch/data

path.logs: /usr/local/elasticsearch/logs

network.host: 192.168.1.103

http.port: 9200

discovery.zen.ping.unicast.hosts:["192.168.1.103","192.168.1.104","192.168.1.105"]

 

说明:

discovery.zen.ping.unicast.hosts:["192.168.1.103","192.168.1.104","192.168.1.105"]es集群的说明,如果是单各es节点的话就没必要配置这个。


2.3kibana配置

# vi /usr/local/kibana/config/kibana.yml

server.port: 5601

 

host: "192.168.1.103"

elasticsearch_url: http://192.168.1.103:9200192.168.1.104:9200,192.168.1.105:9200

 

说明:

elasticsearch_url: http://192.168.1.103:9200192.168.1.104:9200,192.168.1.105:9200也是只想的es集群,如果是单台则只需配置单个es节点。


2.4kafkazookeeper非分离的配置

单个kafka&zookeeper节点

/usr/local/kafka # vi config/zookeeper.properties

dataDir=/usr/local/kafka/tmp/zookeeper

 

/usr/local/kafka # viconfig/server.properties

log.dirs=/usr/local/kafka/tmp/kafka-logs

 

Zookeepe集群配置

# vi config/zookeeper.properties

dataDir=/usr/local/kafka/tmp/zookeeper

initLimit=5

syncLimit=2

server.2=192.168.1.101:2888:3888

server.3=192.168.1.102:2888:3888

server.4=192.168.1.103:2888:3888


kafka集群:

/usr/local/kafka # viconfig/server.properties

broker.id=2

prot=9092

host.name=192.168.1.101

log.dirs=/usr/local/kafka/tmp/kafka-logs

num.partitions=16

zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181

注意其他两个节点broker.id分别是34,host.name也按实际的配置

 

启动zookeeper服务

/usr/local/kafka #./bin/zookeeper-server-start.sh config/zookeeper.properties 


启动kafka服务

/usr/local/kafka #./bin/kafka-server-start.sh config/server.properties 


2.5kafkazookeeper分离的配置

Zookeeper的配置:

生成zookeeper配置文件

# cd zookeeper/conf

# cp zoo_sample.cfg zoo.cfg

 

编辑配置文件

# vi zoo.cfg

dataDir=/usr/local/zookeeper/tmp/zookeeper

server.1=192.168.1.101:2888:3888 

server.2=192.168.1.102:2888:3888 

server.3=192.168.1.103:2888:3888

 

# cd ..

# mkdir -p tmp/zookeeper

# echo "1" >tmp/zookeeper/myid

 

配置node2node3zookeeper

依照node1的配置配置node2node3,注意下面的参数三个节点各有差异

Node2

# echo "2" >tmp/zookeeper/myid

 

Node3

# echo "3" >tmp/zookeeper/myid

 

其他配置都一样

 

依次启动三个节点的服务

# ./bin/zkServer.sh start conf/zoo.cfg

 

Kafka的配置:

配置node1kafka

# cd ../kafka

# vi config/server.properties

broker.id=0

port=9092 

host.name=x-shcs-creditcard-v01

log.dirs=/usr/local/kafka/tmp/kafka-logs

num.partitions=2

zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181

 

配置Node2node3kafka

依照node1的配置配置node2node3,注意下面的参数三个节点各有差异

Node2:

broker.id=1

host.name=node2

 

node3

broker.id=:2

host.name=node3

 

说明:

host.name是节点的主机名

 

依次启动三个节点的kafka

# ./bin/kafka-server-start config/server.properties


三、问题汇总

3.1marvel插件安装出错

node3:/usr/local/elasticsearch# ./bin/plugin install file:///home/apps/license-2.3.3.zip

-> Installing fromfile:/home/apps/license-2.3.3.zip...

Trying file:/home/apps/license-2.3.3.zip...

Downloading .DONE

Verifying file:/home/apps/license-2.3.3.zipchecksums if available ...

NOTE: Unable to verify checksum fordownloaded plugin (unable to find .sha1 or .md5 file to verify)

ERROR: Plugin [license] is incompatiblewith Elasticsearch [2.1.1]. Was designed for version [2.3.3]

 

插件版本和ES版本不兼容,换为2.3.3版本的插件就没有问题了。


3.2、启动es服务出错


/usr/local/elasticsearch>./bin/elasticsearch

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException:/usr/local/elasticsearch/logs/es_cluster.log (Permission denied)

 

文件权限的问题,es服务启动需要切换到非root用户,而该用户没有访问/usr/local/elasticsearch/logs/es_cluster.log的权限。

 

修改文件权限就好了

# chown -R esuser:esuser/usr/local/elasticsearch/data/

# chown -R esuser:esuser/usr/local/elasticsearch/logs


3.3、kibana版本不兼容

# ./bin/kibana 

{"name":"Kibana","hostname":"node3","pid":20969,"level":50,"err":{"message":"unknownerror","name":"Error","stack":"Error:unknown error\n    at respond(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:237:15)\n    at checkRespForFailure(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n    at HttpConnector.<anonymous>(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n    at IncomingMessage.bound(/home/apps/kibana-4.1.4-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n    at IncomingMessage.emit(events.js:117:20)\n    at_stream_readable.js:944:16\n    atprocess._tickCallback (node.js:442:13)"},"msg":"","time":"2016-08-30T07:07:57.923Z","v":0}

 

换为更高的kibana版本就没问题了


3.4、nginx重载配置问价出错

/usr/local/nginx/sbin # ./nginx -s reload

nginx: [error] open()"/usr/local/nginx/logs/nginx.pid" failed (2: No such file ordirectory)

 

/usr/local/nginx/sbin # ls ../logs/

access.log error.log  

 

/usr/local/nginx/sbin # ./nginx -c/usr/local/nginx/conf/nginx.conf

 

/usr/local/nginx/sbin # ls ../logs/

access.log error.log   nginx.pid   


3.5、启动kafka出错有一个broker没起来,提示Failed to acquire lock on file .lock in/usr/local/kafka/tmp/kafka-logs.

关闭kafka服务,查看还有哪个节点上的kafka还有进程,杀死改进城之后再次启动就没问题了。


3.6、kafka删除topic出错,删除不了(还未解决)

/usr/local/kafka # ./bin/kafka-topics.sh--delete --zookeeper 192.168.1.101:2181 --topic nginx-messages

Topic nginx-messages is marked for deletion.

Note: This will have no impact ifdelete.topic.enable is not set to true

 

按网上说的修改

配置文件在kafka\config目录

# vi server.properties  

delete.topic.enable=true

 

但是修改之后还是删除不了topic


四、 Logstash配置文件

4.1、配置一

input {

 stdin {

   type => "stdin-type"

  }

 file {

   type => "syslog-ng"

 

    #Wildcards work, here :)

   path => [ "/var/log/*.log", "/var/log/messages","/var/log/syslog" ]

  }

}

output {

 stdout { }

elasticsearch{

 hosts=>["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]

   }


配置一的架构是被收集日志的机器直接通过本机的logstash把日志传送给es集群


4.2、配置二

input {

file {

type => "system-message"

path => "/var/log/messages"

start_position => "beginning"

}

}

 

output {

#stdout { codec => rubydebug }

kafka {

bootstrap_servers =>"192.168.1.103:9092"

topic_id => "system-messages"

compression_type => "snappy"

}

}

配置二的架构是被收集日志的机器直接通过本机的logstash把日志传送给es单个节点


4.3、配置三

input {

kafka {

zk_connect =>"192.168.1.103:9092"

topic_id => "System-Messages"

codec => plain

reset_beginning => false

consumer_threads => 5

decorate_events => true

}

}

 

output {

elasticsearch {

hosts => "192.168.1.103:9092"

index =>"test-System-Messages-%{+YYYY-MM}"

}

}

配置三的架构是被kafka节点收集到日志之后传送给es节点


4.4、配置四

# vi/usr/local/logstash/etc/logstash_shipper.conf

input {

file {

type => "system-message"

path => "/var/log/messages"

start_position => "beginning"

}

}

output {

stdout { codec => rubydebug }

kafka {

bootstrap_servers =>"192.168.1.101:9092,192.168.1.102:9092,192.168.1.103:9092"

topic_id => "messages"

compression_type => "snappy"

}


logstash消费端

/usr/local/logstash # vietc/logstash_indexer.conf

input {

kafka {

zk_connect => "192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181"

topic_id => "system-message"

codec => plain

reset_beginning => false

consumer_threads => 5

decorate_events => true

}

}

output {

elasticsearch {

hosts => "192.168.1.105:9200"

index =>"test-system-messages-%{+YYYY-MM}"

}

}

配置四的架构是被收集日志的机器直接通过本机的logstash把日志传送给kafka集群,之后kafka集群再把日志传送给es节点。


本文出自 “xiaoxiaozhou” 博客,请务必保留此出处http://xiaoxiaozhou.blog.51cto.com/4681537/1855188

ELK配置总结