首页 > 代码库 > logstash

logstash

logstash作为数据搜集器,主要分为三个部分:input->filter->output  作为pipeline的形式进行处理,支持复杂的操作,如发邮件等

   input配置数据的输入和简单的数据转换

  filter配置数据的提取,一般使用grok

  output配置数据的输出和简单的数据转换

  运行:logstash  -f /etc/logstash.conf

            -f  指定配置文件

           -e  只在控制台运行

具体的配置见官网

https://www.elastic.co/products/logstash

Centralize, Transform & Stash Your Data

    input

Plugin

Description

Github repository

beats

Receives events from the Elastic Beats framework

logstash-input-beats

couchdb_changes

Streams events from CouchDB’s _changes URI

logstash-input-couchdb_changes

elasticsearch

Reads query results from an Elasticsearch cluster

logstash-input-elasticsearch

file

Streams events from files

logstash-input-file

gelf

Reads GELF-format messages from Graylog2 as events

logstash-input-gelf

generator

Generates random log events for test purposes

logstash-input-generator

graphite

Reads metrics from the graphite tool

logstash-input-graphite

heartbeat

Generates heartbeat events for testing

logstash-input-heartbeat

http

Receives events over HTTP or HTTPS

logstash-input-http

http_poller

Decodes the output of an HTTP API into events

logstash-input-http_poller

jdbc

Creates events from JDBC data

logstash-input-jdbc

kafka

Reads events from a Kafka topic

logstash-input-kafka

log4j

Reads events over a TCP socket from a Log4jSocketAppender object

logstash-input-log4j

lumberjack

Receives events using the Lumberjack protocl

logstash-input-lumberjack

rabbitmq

Pulls events from a RabbitMQ exchange

logstash-input-rabbitmq

redis

Reads events from a Redis instance

logstash-input-redis

s3

Streams events from files in a S3 bucket

logstash-input-s3

sqs

Pulls events from an Amazon Web Services Simple Queue Service queue

logstash-input-sqs

stdin

Reads events from standard input

logstash-input-stdin

syslog

Reads syslog messages as events

logstash-input-syslog

tcp

Reads events from a TCP socket

logstash-input-tcp

twitter

Reads events from the Twitter Streaming API

logstash-input-twitter

udp

Reads events over UDP

logstash-input-udp

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin

Description

Github repository

cloudwatch

Pulls events from the Amazon Web Services CloudWatch API

logstash-input-cloudwatch

drupal_dblog

Retrieves watchdog log events from Drupal installations with DBLog enabled

logstash-input-drupal_dblog

eventlog

Pulls events from the Windows Event Log

logstash-input-eventlog

exec

Captures the output of a shell command as an event

logstash-input-exec

ganglia

Reads Ganglia packets over UDP

logstash-input-ganglia

gemfire

Pushes events to a GemFire region

logstash-input-gemfire

github

Reads events from a GitHub webhook

logstash-input-github

heroku

Streams events from the logs of a Heroku app

logstash-input-heroku

imap

Reads mail from an IMAP server

logstash-input-imap

irc

Reads events from an IRC server

logstash-input-irc

jmx

Retrieves metrics from remote Java applications over JMX

logstash-input-jmx

kinesis

Receives events through an AWS Kinesis stream

logstash-input-kinesis

meetup

Captures the output of command line tools as an event

logstash-input-meetup

pipe

Streams events from a long-running command pipe

logstash-input-pipe

puppet_facter

Receives facts from a Puppet server

logstash-input-puppet_facter

rackspace

Receives events from a Rackspace Cloud Queue service

logstash-input-rackspace

relp

Receives RELP events over a TCP socket

logstash-input-relp

rss

Captures the output of command line tools as an event

logstash-input-rss

salesforce

Creates events based on a Salesforce SOQL query

logstash-input-salesforce

snmptrap

Creates events based on SNMP trap messages

logstash-input-snmptrap

sqlite

Creates events based on rows in an SQLite database

logstash-input-sqlite

stomp

Creates events received with the STOMP protocol

logstash-input-stomp

unix

Reads events over a UNIX socket

logstash-input-unix

varnishlog

Reads from the varnish cache shared memory log

logstash-input-varnishlog

websocket

Reads events from a websocket

logstash-input-websocket

wmi

Creates events based on the results of a WMI query

logstash-input-wmi

xmpp

Receives events over the XMPP/Jabber protocol

logstash-input-xmpp

zenoss

Reads Zenoss events from the fanout exchange

logstash-input-zenoss

zeromq

Reads events from a ZeroMQ SUB socket

logstash-input-zeromq

 

filter

Plugin

Description

Github repository

aggregate

Aggregates information from several events originating with a single task

logstash-filter-aggregate

anonymize

Replaces field values with a consistent hash

logstash-filter-anonymize

csv

Parses comma-separated value data into individual fields

logstash-filter-csv

date

Parses dates from fields to use as the Logstash timestamp for an event

logstash-filter-date

de_dot

Computationally expensive filter that removes dots from a field name

logstash-filter-de_dot

dissect

Extracts unstructured event data into fields using delimiters

logstash-filter-dissect

dns

Performs a standard or reverse DNS lookup

logstash-filter-dns

drop

Drops all events

logstash-filter-drop

fingerprint

Fingerprints fields by replacing values with a consistent hash

logstash-filter-fingerprint

geoip

Adds geographical information about an IP address

logstash-filter-geoip

grok

Parses unstructured event data into fields

logstash-filter-grok

json

Parses JSON events

logstash-filter-json

kv

Parses key-value pairs

logstash-filter-kv

multiline

Merges multiple lines into a single event

logstash-filter-multiline

mutate

Performs mutations on fields

logstash-filter-mutate

ruby

Executes arbitrary Ruby code

logstash-filter-ruby

sleep

Sleeps for a specified time span

logstash-filter-sleep

split

Splits multi-line messages into distinct events

logstash-filter-split

syslog_pri

Parses the PRI (priority) field of a syslog message

logstash-filter-syslog_pri

throttle

Throttles the number of events

logstash-filter-throttle

translate

Replaces field contents based on a hash or YAML file

logstash-filter-translate

urldecode

Decodes URL-encoded fields

logstash-filter-urldecode

useragent

Parses user agent strings into fields

logstash-filter-useragent

uuid

Adds a UUID to events

logstash-filter-uuid

xml

Parses XML into fields

logstash-filter-xml

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin

Description

Github repository

alter

Performs general alterations to fields that the mutate filter does not handle

logstash-filter-alter

cidr

Checks IP addresses against a list of network blocks

logstash-filter-cidr

cipher

Applies or removes a cipher to an event

logstash-filter-cipher

clone

Duplicates events

logstash-filter-clone

collate

Collates events by time or count

logstash-filter-collate

elapsed

Calculates the elapsed time between a pair of events

logstash-filter-elapsed

elasticsearch

Copies fields from previous log events in Elasticsearch to current events

logstash-filter-elasticsearch

environment

Stores environment variables as metadata sub-fields

logstash-filter-environment

extractnumbers

Extracts numbers from a string

logstash-filter-extractnumbers

i18n

Removes special characters from a field

logstash-filter-i18n

json_encode

Serializes a field to JSON

logstash-filter-json_encode

metaevent

Adds arbitrary fields to an event

logstash-filter-metaevent

metricize

Takes complex events containing a number of metrics and splits these up into multiple events, each holding a single metric

logstash-filter-metricize

metrics

Aggregates metrics

logstash-filter-metrics

oui

Parse OUI data from MAC addresses

logstash-filter-oui

prune

Prunes event data based on a list of fields to blacklist or whitelist

logstash-filter-prune

punct

Strips all non-punctuation content from a field

logstash-filter-punct

range

Checks that specified fields stay within given size or length limits

logstash-filter-range

tld

Replaces the contents of the default message field with whatever you specify in the configuration

logstash-filter-tld

yaml

Takes an existing field that contains YAML and expands it into an actual data structure within the Logstash event

logstash-filter-yaml

zeromq

Sends an event to ZeroMQ

logstash-filter-zeromq

 

 

output

Elastic supported plugins

These plugins are maintained and supported by Elastic.

Plugin

Description

Github repository

csv

Writes events to disk in a delimited format

logstash-output-csv

elasticsearch

Stores logs in Elasticsearch

logstash-output-elasticsearch

email

Sends email to a specified address when output is received

logstash-output-email

file

Writes events to files on disk

logstash-output-file

graphite

Writes metrics to Graphite

logstash-output-graphite

http

Sends events to a generic HTTP or HTTPS endpoint

logstash-output-http

kafka

Writes events to a Kafka topic

logstash-output-kafka

lumberjack

Sends events using the lumberjack protocol

logstash-output-lumberjack

rabbitmq

Pushes events to a RabbitMQ exchange

logstash-output-rabbitmq

redis

Sends events to a Redis queue using the RPUSHcommand

logstash-output-redis

s3

Sends Logstash events to the Amazon Simple Storage Service

logstash-output-s3

stdout

Prints events to the standard output

logstash-output-stdout

tcp

Writes events over a TCP socket

logstash-output-tcp

udp

Sends events over UDP

logstash-output-udp

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin

Description

Github repository

boundary

Sends annotations to Boundary based on Logstash events

logstash-output-boundary

circonus

Sends annotations to Circonus based on Logstash events

logstash-output-circonus

cloudwatch

Aggregates and sends metric data to AWS CloudWatch

logstash-output-cloudwatch

datadog

Sends events to DataDogHQ based on Logstash events

logstash-output-datadog

datadog_metrics

Sends metrics to DataDogHQ based on Logstash events

logstash-output-datadog_metrics

elasticsearch_java

Stores logs in Elasticsearch using the node andtransport protocols

logstash-output-elasticsearch_java

exec

Runs a command for a matching event

logstash-output-exec

ganglia

Writes metrics to Ganglia’s gmond

logstash-output-ganglia

gelf

Generates GELF formatted output for Graylog2

logstash-output-gelf

google_bigquery

Writes events to Google BigQuery

logstash-output-google_bigquery

google_cloud_storage

Writes events to Google Cloud Storage

logstash-output-google_cloud_storage

graphtastic

Sends metric data on Windows

logstash-output-graphtastic

hipchat

Writes events to HipChat

logstash-output-hipchat

influxdb

Writes metrics to InfluxDB

logstash-output-influxdb

irc

Writes events to IRC

logstash-output-irc

jira

Writes strutured JSON events to JIRA

logstash-output-jira

juggernaut

Pushes messages to the Juggernaut websockets server

logstash-output-juggernaut

librato

Sends metrics, annotations, and alerts to Librato based on Logstash events

logstash-output-librato

loggly

Ships logs to Loggly

logstash-output-loggly

metriccatcher

Writes metrics to MetricCatcher

logstash-output-metriccatcher

mongodb

Writes events to MongoDB

logstash-output-mongodb

nagios

Sends passive check results to Nagios

logstash-output-nagios

nagios_nsca

Sends passive check results to Nagios using the NSCA protocol

logstash-output-nagios_nsca

newrelic

Sends logstash events to New Relic Insights as custom events

logstash-output-newrelic

opentsdb

Writes metrics to OpenTSDB

logstash-output-opentsdb

pagerduty

Sends notifications based on preconfigured services and escalation policies

logstash-output-pagerduty

pipe

Pipes events to another program’s standard input

logstash-output-pipe

rackspace

Sends events to a Rackspace Cloud Queue service

logstash-output-rackspace

redmine

Creates tickets using the Redmine API

logstash-output-redmine

riak

Writes events to the Riak distributed key/value store

logstash-output-riak

riemann

Sends metrics to Riemann

logstash-output-riemann

sns

Sends events to Amazon’s Simple Notification Service

logstash-output-sns

solr_http

Stores and indexes logs in Solr

logstash-output-solr_http

sqs

Pushes events to an Amazon Web Services Simple Queue Serice queue

logstash-output-sqs

statsd

Sends metrics using the statsd network daemon

logstash-output-statsd

stomp

Writes events using the STOMP protocol

logstash-output-stomp

syslog

Sends events to a syslog server

logstash-output-syslog

webhdfs

Sends Logstash events to HDFS using the webhdfsREST API

logstash-output-webhdfs

websocket

Publishes messages to a websocket

logstash-output-websocket

xmpp

Posts events over XMPP

logstash-output-xmpp

zabbix

Sends events to a Zabbix server

logstash-output-zabbix

zeromq

Writes events to a ZeroMQ PUB socket

logstash-output-zeromq

 

logstash