首页 > 代码库 > logstash
logstash
logstash作为数据搜集器,主要分为三个部分:input->filter->output 作为pipeline的形式进行处理,支持复杂的操作,如发邮件等
input配置数据的输入和简单的数据转换
filter配置数据的提取,一般使用grok
output配置数据的输出和简单的数据转换
运行:logstash -f /etc/logstash.conf
-f 指定配置文件
-e 只在控制台运行
具体的配置见官网
https://www.elastic.co/products/logstash
Centralize, Transform & Stash Your Data
input
Plugin |
Description |
Github repository |
beats |
Receives events from the Elastic Beats framework |
logstash-input-beats |
couchdb_changes |
Streams events from CouchDB’s |
logstash-input-couchdb_changes |
elasticsearch |
Reads query results from an Elasticsearch cluster |
logstash-input-elasticsearch |
file |
Streams events from files |
logstash-input-file |
gelf |
Reads GELF-format messages from Graylog2 as events |
logstash-input-gelf |
generator |
Generates random log events for test purposes |
logstash-input-generator |
graphite |
Reads metrics from the |
logstash-input-graphite |
heartbeat |
Generates heartbeat events for testing |
logstash-input-heartbeat |
http |
Receives events over HTTP or HTTPS |
logstash-input-http |
http_poller |
Decodes the output of an HTTP API into events |
logstash-input-http_poller |
jdbc |
Creates events from JDBC data |
logstash-input-jdbc |
kafka |
Reads events from a Kafka topic |
logstash-input-kafka |
log4j |
Reads events over a TCP socket from a Log4j |
logstash-input-log4j |
lumberjack |
Receives events using the Lumberjack protocl |
logstash-input-lumberjack |
rabbitmq |
Pulls events from a RabbitMQ exchange |
logstash-input-rabbitmq |
redis |
Reads events from a Redis instance |
logstash-input-redis |
s3 |
Streams events from files in a S3 bucket |
logstash-input-s3 |
sqs |
Pulls events from an Amazon Web Services Simple Queue Service queue |
logstash-input-sqs |
stdin |
Reads events from standard input |
logstash-input-stdin |
syslog |
Reads syslog messages as events |
logstash-input-syslog |
tcp |
Reads events from a TCP socket |
logstash-input-tcp |
|
Reads events from the Twitter Streaming API |
logstash-input-twitter |
udp |
Reads events over UDP |
logstash-input-udp |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
Plugin |
Description |
Github repository |
cloudwatch |
Pulls events from the Amazon Web Services CloudWatch API |
logstash-input-cloudwatch |
drupal_dblog |
Retrieves watchdog log events from Drupal installations with DBLog enabled |
logstash-input-drupal_dblog |
eventlog |
Pulls events from the Windows Event Log |
logstash-input-eventlog |
exec |
Captures the output of a shell command as an event |
logstash-input-exec |
ganglia |
Reads Ganglia packets over UDP |
logstash-input-ganglia |
gemfire |
Pushes events to a GemFire region |
logstash-input-gemfire |
github |
Reads events from a GitHub webhook |
logstash-input-github |
heroku |
Streams events from the logs of a Heroku app |
logstash-input-heroku |
imap |
Reads mail from an IMAP server |
logstash-input-imap |
irc |
Reads events from an IRC server |
logstash-input-irc |
jmx |
Retrieves metrics from remote Java applications over JMX |
logstash-input-jmx |
kinesis |
Receives events through an AWS Kinesis stream |
logstash-input-kinesis |
meetup |
Captures the output of command line tools as an event |
logstash-input-meetup |
pipe |
Streams events from a long-running command pipe |
logstash-input-pipe |
puppet_facter |
Receives facts from a Puppet server |
logstash-input-puppet_facter |
rackspace |
Receives events from a Rackspace Cloud Queue service |
logstash-input-rackspace |
relp |
Receives RELP events over a TCP socket |
logstash-input-relp |
rss |
Captures the output of command line tools as an event |
logstash-input-rss |
salesforce |
Creates events based on a Salesforce SOQL query |
logstash-input-salesforce |
snmptrap |
Creates events based on SNMP trap messages |
logstash-input-snmptrap |
sqlite |
Creates events based on rows in an SQLite database |
logstash-input-sqlite |
stomp |
Creates events received with the STOMP protocol |
logstash-input-stomp |
unix |
Reads events over a UNIX socket |
logstash-input-unix |
varnishlog |
Reads from the |
logstash-input-varnishlog |
websocket |
Reads events from a websocket |
logstash-input-websocket |
wmi |
Creates events based on the results of a WMI query |
logstash-input-wmi |
xmpp |
Receives events over the XMPP/Jabber protocol |
logstash-input-xmpp |
zenoss |
Reads Zenoss events from the fanout exchange |
logstash-input-zenoss |
zeromq |
Reads events from a ZeroMQ SUB socket |
logstash-input-zeromq |
filter
Plugin |
Description |
Github repository |
aggregate |
Aggregates information from several events originating with a single task |
logstash-filter-aggregate |
anonymize |
Replaces field values with a consistent hash |
logstash-filter-anonymize |
csv |
Parses comma-separated value data into individual fields |
logstash-filter-csv |
date |
Parses dates from fields to use as the Logstash timestamp for an event |
logstash-filter-date |
de_dot |
Computationally expensive filter that removes dots from a field name |
logstash-filter-de_dot |
dissect |
Extracts unstructured event data into fields using delimiters |
logstash-filter-dissect |
dns |
Performs a standard or reverse DNS lookup |
logstash-filter-dns |
drop |
Drops all events |
logstash-filter-drop |
fingerprint |
Fingerprints fields by replacing values with a consistent hash |
logstash-filter-fingerprint |
geoip |
Adds geographical information about an IP address |
logstash-filter-geoip |
grok |
Parses unstructured event data into fields |
logstash-filter-grok |
json |
Parses JSON events |
logstash-filter-json |
kv |
Parses key-value pairs |
logstash-filter-kv |
multiline |
Merges multiple lines into a single event |
logstash-filter-multiline |
mutate |
Performs mutations on fields |
logstash-filter-mutate |
ruby |
Executes arbitrary Ruby code |
logstash-filter-ruby |
sleep |
Sleeps for a specified time span |
logstash-filter-sleep |
split |
Splits multi-line messages into distinct events |
logstash-filter-split |
syslog_pri |
Parses the |
logstash-filter-syslog_pri |
throttle |
Throttles the number of events |
logstash-filter-throttle |
translate |
Replaces field contents based on a hash or YAML file |
logstash-filter-translate |
urldecode |
Decodes URL-encoded fields |
logstash-filter-urldecode |
useragent |
Parses user agent strings into fields |
logstash-filter-useragent |
uuid |
Adds a UUID to events |
logstash-filter-uuid |
xml |
Parses XML into fields |
logstash-filter-xml |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
Plugin |
Description |
Github repository |
alter |
Performs general alterations to fields that the |
logstash-filter-alter |
cidr |
Checks IP addresses against a list of network blocks |
logstash-filter-cidr |
cipher |
Applies or removes a cipher to an event |
logstash-filter-cipher |
clone |
Duplicates events |
logstash-filter-clone |
collate |
Collates events by time or count |
logstash-filter-collate |
elapsed |
Calculates the elapsed time between a pair of events |
logstash-filter-elapsed |
elasticsearch |
Copies fields from previous log events in Elasticsearch to current events |
logstash-filter-elasticsearch |
environment |
Stores environment variables as metadata sub-fields |
logstash-filter-environment |
extractnumbers |
Extracts numbers from a string |
logstash-filter-extractnumbers |
i18n |
Removes special characters from a field |
logstash-filter-i18n |
json_encode |
Serializes a field to JSON |
logstash-filter-json_encode |
metaevent |
Adds arbitrary fields to an event |
logstash-filter-metaevent |
metricize |
Takes complex events containing a number of metrics and splits these up into multiple events, each holding a single metric |
logstash-filter-metricize |
metrics |
Aggregates metrics |
logstash-filter-metrics |
oui |
Parse OUI data from MAC addresses |
logstash-filter-oui |
prune |
Prunes event data based on a list of fields to blacklist or whitelist |
logstash-filter-prune |
punct |
Strips all non-punctuation content from a field |
logstash-filter-punct |
range |
Checks that specified fields stay within given size or length limits |
logstash-filter-range |
tld |
Replaces the contents of the default message field with whatever you specify in the configuration |
logstash-filter-tld |
yaml |
Takes an existing field that contains YAML and expands it into an actual data structure within the Logstash event |
logstash-filter-yaml |
zeromq |
Sends an event to ZeroMQ |
logstash-filter-zeromq |
output
Elastic supported plugins
These plugins are maintained and supported by Elastic.
Plugin |
Description |
Github repository |
csv |
Writes events to disk in a delimited format |
logstash-output-csv |
elasticsearch |
Stores logs in Elasticsearch |
logstash-output-elasticsearch |
|
Sends email to a specified address when output is received |
logstash-output-email |
file |
Writes events to files on disk |
logstash-output-file |
graphite |
Writes metrics to Graphite |
logstash-output-graphite |
http |
Sends events to a generic HTTP or HTTPS endpoint |
logstash-output-http |
kafka |
Writes events to a Kafka topic |
logstash-output-kafka |
lumberjack |
Sends events using the |
logstash-output-lumberjack |
rabbitmq |
Pushes events to a RabbitMQ exchange |
logstash-output-rabbitmq |
redis |
Sends events to a Redis queue using the |
logstash-output-redis |
s3 |
Sends Logstash events to the Amazon Simple Storage Service |
logstash-output-s3 |
stdout |
Prints events to the standard output |
logstash-output-stdout |
tcp |
Writes events over a TCP socket |
logstash-output-tcp |
udp |
Sends events over UDP |
logstash-output-udp |
Community supported plugins
These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.
Plugin |
Description |
Github repository |
boundary |
Sends annotations to Boundary based on Logstash events |
logstash-output-boundary |
circonus |
Sends annotations to Circonus based on Logstash events |
logstash-output-circonus |
cloudwatch |
Aggregates and sends metric data to AWS CloudWatch |
logstash-output-cloudwatch |
datadog |
Sends events to DataDogHQ based on Logstash events |
logstash-output-datadog |
datadog_metrics |
Sends metrics to DataDogHQ based on Logstash events |
logstash-output-datadog_metrics |
elasticsearch_java |
Stores logs in Elasticsearch using the |
logstash-output-elasticsearch_java |
exec |
Runs a command for a matching event |
logstash-output-exec |
ganglia |
Writes metrics to Ganglia’s |
logstash-output-ganglia |
gelf |
Generates GELF formatted output for Graylog2 |
logstash-output-gelf |
google_bigquery |
Writes events to Google BigQuery |
logstash-output-google_bigquery |
google_cloud_storage |
Writes events to Google Cloud Storage |
logstash-output-google_cloud_storage |
graphtastic |
Sends metric data on Windows |
logstash-output-graphtastic |
hipchat |
Writes events to HipChat |
logstash-output-hipchat |
influxdb |
Writes metrics to InfluxDB |
logstash-output-influxdb |
irc |
Writes events to IRC |
logstash-output-irc |
jira |
Writes strutured JSON events to JIRA |
logstash-output-jira |
juggernaut |
Pushes messages to the Juggernaut websockets server |
logstash-output-juggernaut |
librato |
Sends metrics, annotations, and alerts to Librato based on Logstash events |
logstash-output-librato |
loggly |
Ships logs to Loggly |
logstash-output-loggly |
metriccatcher |
Writes metrics to MetricCatcher |
logstash-output-metriccatcher |
mongodb |
Writes events to MongoDB |
logstash-output-mongodb |
nagios |
Sends passive check results to Nagios |
logstash-output-nagios |
nagios_nsca |
Sends passive check results to Nagios using the NSCA protocol |
logstash-output-nagios_nsca |
newrelic |
Sends logstash events to New Relic Insights as custom events |
logstash-output-newrelic |
opentsdb |
Writes metrics to OpenTSDB |
logstash-output-opentsdb |
pagerduty |
Sends notifications based on preconfigured services and escalation policies |
logstash-output-pagerduty |
pipe |
Pipes events to another program’s standard input |
logstash-output-pipe |
rackspace |
Sends events to a Rackspace Cloud Queue service |
logstash-output-rackspace |
redmine |
Creates tickets using the Redmine API |
logstash-output-redmine |
riak |
Writes events to the Riak distributed key/value store |
logstash-output-riak |
riemann |
Sends metrics to Riemann |
logstash-output-riemann |
sns |
Sends events to Amazon’s Simple Notification Service |
logstash-output-sns |
solr_http |
Stores and indexes logs in Solr |
logstash-output-solr_http |
sqs |
Pushes events to an Amazon Web Services Simple Queue Serice queue |
logstash-output-sqs |
statsd |
Sends metrics using the |
logstash-output-statsd |
stomp |
Writes events using the STOMP protocol |
logstash-output-stomp |
syslog |
Sends events to a |
logstash-output-syslog |
webhdfs |
Sends Logstash events to HDFS using the |
logstash-output-webhdfs |
websocket |
Publishes messages to a websocket |
logstash-output-websocket |
xmpp |
Posts events over XMPP |
logstash-output-xmpp |
zabbix |
Sends events to a Zabbix server |
logstash-output-zabbix |
zeromq |
Writes events to a ZeroMQ PUB socket |
logstash-output-zeromq |
logstash