首页 > 代码库 > saltstack结合Elasticsearch来做salt运行结果展现
saltstack结合Elasticsearch来做salt运行结果展现
salt尽管好用可是机器管理的越来越多,通过cli的结果输出方式查看运行结果越来越多不能满足我的需求。并且作为一个推动运维自己主动化的攻城狮,使用这样的人眼查看运行结果的方式简直土到掉渣。尽管别人看起来逼格非常高。但谁累谁知道。。。因为以上原因,给各位推荐一种逼格更高的结果查看方式:
salt returners
先来看一下官方结构图:
Send data returned by Salt Minions to another system, such as a database. Returners can run on the Salt Minion or on the Salt Master.
这个图上画的是redis,事实上官方给出的returners有非常多非常多。这次为了装逼。哦不正确。为了更好地筛选和过滤salt的运行结果,我选用了elasticsearch+kibana这样的展现方式,优点是es提供一个准实时的全文检索引擎。这样我们能够实时的想搜什么搜什么,多维度过滤,而kibana提供的是便捷的用户界面和帅的一逼的绘图界面!
可是。这两者是怎么结合起来的呢,非常easy,用一个es支持salt也支持的数据中转站即可了。我选择的是kafka。当然你也能够用rsyslog。redis,等等,任君选择
以下就以 salt-minion -> kafka -> logstash -> elasticseach <- kibana <- 你的浏览器 这条主线进行说明,有纰漏的地方欢迎在评论里指正,谢谢!
如今张成果图镇楼:
以下正时開始:
1. salt-minion端的配置:
1.1 改动minion配置文件,这对于会用salt的我们来说改1w台机器就是分分钟的事儿嘛。但注意。改完要重新启动minion进程。
/etc/salt/minion:
#kafka broker主机,能够配置多个 returner.kafka.hostnames: - "10.64.0.1" - "10.64.0.2" - "10.64.0.3" #kafka topic名称 returner.kafka.topic: 'saltstack-topic'
1.2 改动官方returner文件源码,此处高深莫測。先按下不表
2.kafka 差点儿不用配置。正常能用就可以。顺便推荐一下kafka manager真好用。从此再也不用记命令了。
https://github.com/yahoo/kafka-manager
3.logstash 作为刚才minion发送到kafka数据的消费者,拿到kafka的数据后按json格式存到es中
3.1 logstash.conf:
input { kafka { #zk 连接串 zk_connect => '10.64.0.1:2181,10.64.0.2:2181,10.64.0.3:2181,10.64.1.174:2181,10.64.1.175:2181' #与刚才配置的topic一致 group_id => 'logstash-saltstack' topic_id => 'saltstack-topic' consumer_id => 'logstash-saltstack-consumer-{{ grains["id"] }}' consumer_threads => 1 queue_size => 200 codec => plain } } filter { mutate { add_field => { "fromAgent" => "logstash-saltstack-{{ grains["id"] }}" } } json { source => "message" } mutate { remove_field => [ "message" ] convert => { "fun_args" => "string" } rename => { "__run_num__" => "run_num" } } } output { elasticsearch { #输出到es hosts => ["10.64.0.1:9200","10.64.0.2:9200","10.64.0.3:9200"] index => "saltstack-%{+YYYY.MM}" flush_size => 3000 idle_flush_time => 2 workers => 2 } #stdout { codec => rubydebug } }
这里有个小坑,顺便解释一下刚才按下不表的为啥非要改源代码:es收数据的时候会把json全都拆成嵌套的kv格式存储,啥意思呢,请看图:
这是两个命令运行的不同返回,问题在哪儿呢?问题就在10.64.0.1这个key的value。一会儿是boolen型一会儿是nested型,这样es就没法办了。一个字段的类型实在mapping或者第一次创建数据时确定的。不能随便更改,这样就造成了es不能接收数据的问题。
怎么解决?開始我尝试了改动logstash的配置,只是那个实在不灵活。并且你注意看第二个salt运行的结果,是在一个大json里面的,这样我搜索起来匹配上的话也是一整条数据。还是不方便我看结果。
针对以上两点我还是认为小改一下salt的returners,改动的地方例如以下(每台minion都要改。只是这次不用重新启动了):
/usr/lib/python2.6/site-packages/salt/returners/kafka_return.py:
def returner(ret): ''' Return information to a Kafka server ''' if __salt__['config.option']('returner.kafka.topic'): topic = __salt__['config.option']('returner.kafka.topic') conn = _get_conn(ret) producer = SimpleProducer(conn) if ret["return"] == True or ret["return"] == False: producer.send_messages(topic, json.dumps(ret)) else: for retKey in ret["return"]: myRet = {} myRet["id"] = ret["id"] myRet["fun"] = ret["fun"] myRet["fun_args"] = ret["fun_args"] myRet["jid"] = ret["jid"] myRet["retcode"] = ret["retcode"] for key in ret["return"][retKey]: myRet[key] = ret["return"][retKey][key] retKeys = retKey.split("_|-") for i in range(0,len(retKeys)): myRet["key"+str(i)] = retKeys[i] producer.send_messages(topic, json.dumps(myRet)) _close_conn(conn) else: log.error('Unable to find kafka returner config option: topic')
至此我们就应该能在es里看到每一个动作一条记录的salt结果了。
4. es 还是没啥可配置的,正常运转就能够
5. kibana
首先。我用的是kibana 3,这个须要es <2.0版本号的配合,不要赶新潮装es2,这样就用不了我的模板啦
5.1 加载kibana模板
把以下的内容存成一个文件,然后在kibana中选择加载。然后赶紧跑个salt任务试试
神马?没有,是这样。我忘了跟你说以后跑任务的时候要加个參数才干在es里看到哦:
salt 你要运行的命令 --return kafka
这样你就得到了開始看到的那个激动人心的逼格界面。 我有啥没说清楚的欢迎在以下评论里写明,我会尽力解答。谢谢!
{ "title": "Saltstack returns", "services": { "query": { "idQueue": [ 1 ], "list": { "0": { "query": "result:\"true\"", "alias": "成功", "color": "#7EB26D", "id": 0, "pin": false, "type": "lucene", "enable": true }, "4": { "id": 4, "color": "#E24D42", "alias": "失败", "pin": false, "type": "lucene", "enable": true, "query": "result:\"false\"" } }, "ids": [ 0, 4 ] }, "filter": { "idQueue": [ 1 ], "list": { "0": { "type": "time", "field": "@timestamp", "from": "now-1h", "to": "now", "mandate": "must", "active": true, "alias": "", "id": 0 } }, "ids": [ 0 ] } }, "rows": [ { "title": "分类统计", "height": "367px", "editable": true, "collapse": true, "collapsable": true, "panels": [ { "error": false, "span": 4, "editable": true, "type": "terms", "loadingEditor": false, "field": "type", "exclude": [], "missing": false, "other": false, "size": 50, "order": "count", "style": { "font-size": "10pt" }, "donut": false, "tilt": false, "labels": true, "arrangement": "horizontal", "chart": "table", "counter_pos": "below", "spyable": true, "queries": { "mode": "all", "ids": [ 0, 1, 2 ] }, "tmode": "terms", "tstat": "total", "valuefield": "", "title": "type" }, { "error": false, "span": 4, "editable": true, "type": "terms", "loadingEditor": false, "field": "host.raw", "exclude": [], "missing": false, "other": false, "size": 11, "order": "count", "style": { "font-size": "10pt" }, "donut": false, "tilt": false, "labels": true, "arrangement": "horizontal", "chart": "table", "counter_pos": "none", "spyable": true, "queries": { "mode": "all", "ids": [ 0, 1, 2 ] }, "tmode": "terms", "tstat": "total", "valuefield": "", "title": "host" }, { "error": false, "span": 4, "editable": true, "type": "terms", "loadingEditor": false, "field": "tags.raw", "exclude": [], "missing": false, "other": false, "size": 50, "order": "count", "style": { "font-size": "10pt" }, "donut": false, "tilt": false, "labels": true, "arrangement": "horizontal", "chart": "pie", "counter_pos": "none", "spyable": true, "queries": { "mode": "all", "ids": [ 0, 1, 2 ] }, "tmode": "terms", "tstat": "total", "valuefield": "", "title": "tags" } ], "notice": false }, { "title": "Graph", "height": "150px", "editable": true, "collapse": false, "collapsable": true, "panels": [ { "span": 12, "editable": true, "group": [ "default" ], "type": "histogram", "mode": "count", "time_field": "@timestamp", "value_field": null, "auto_int": true, "resolution": 200, "interval": "30s", "fill": 3, "linewidth": 3, "timezone": "browser", "spyable": true, "zoomlinks": true, "bars": true, "stack": true, "points": false, "lines": false, "legend": true, "x-axis": true, "y-axis": true, "percentage": false, "interactive": true, "queries": { "mode": "all", "ids": [ 0, 4 ] }, "title": "Events over time", "intervals": [ "auto", "1s", "1m", "5m", "10m", "30m", "1h", "3h", "12h", "1d", "1w", "1M", "1y" ], "options": true, "tooltip": { "value_type": "cumulative", "query_as_alias": true }, "annotate": { "enable": false, "query": "*", "size": 20, "field": "_type", "sort": [ "_score", "desc" ] }, "pointradius": 5, "show_query": true, "legend_counts": true, "zerofill": true, "derivative": false, "scale": 1, "grid": { "max": null, "min": 0 }, "y_format": "none" } ], "notice": false }, { "title": "Events", "height": "350px", "editable": true, "collapse": false, "collapsable": true, "panels": [ { "title": "All events", "error": false, "span": 12, "editable": true, "group": [ "default" ], "type": "table", "size": 100, "pages": 5, "offset": 0, "sort": [ "@timestamp", "desc" ], "style": { "font-size": "9pt" }, "overflow": "min-height", "fields": [ "@timestamp", "id", "fun", "fun_args", "key0", "key1", "key2", "key3", "result" ], "highlight": [], "sortable": true, "header": true, "paging": true, "spyable": true, "queries": { "mode": "all", "ids": [ 0, 4 ] }, "field_list": true, "status": "Stable", "trimFactor": 300, "normTimes": true, "all_fields": false, "localTime": true, "timeField": "@timestamp" } ], "notice": false } ], "editable": true, "failover": false, "index": { "interval": "month", "pattern": "[saltstack-]YYYY.MM", "default": "NO_TIME_FILTER_OR_INDEX_PATTERN_NOT_MATCHED", "warm_fields": true }, "style": "dark", "panel_hints": true, "pulldowns": [ { "type": "query", "collapse": true, "notice": false, "query": "*", "pinned": true, "remember": 10, "enable": true }, { "type": "filtering", "collapse": true, "notice": false, "enable": true } ], "nav": [ { "type": "timepicker", "collapse": false, "notice": false, "status": "Stable", "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ], "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "timefield": "@timestamp", "now": true, "filter_id": 0, "enable": true } ], "loader": { "save_gist": false, "save_elasticsearch": true, "save_local": true, "save_default": true, "save_temp": true, "save_temp_ttl_enable": true, "save_temp_ttl": "30d", "load_gist": true, "load_elasticsearch": true, "load_elasticsearch_size": 20, "load_local": true, "hide": false }, "refresh": false }
saltstack结合Elasticsearch来做salt运行结果展现