Logstash Integration
Logstash is a free and open server-side data processing pipeline that allows you to collect data from a variety of sources and ingests it to your favorite destination.
Wavefront supports two methods for sending log data to the Wavefront proxy:
- Filebeat: Once your data arrives at the proxy, the proxy converts your Logstash log data to metrics by parsing log lines with grok patterns (regular expressions) that you specify in a proxy configuration file.
- Wavefront Output Plugin: Wavefront Output Plugin for Logstash parses the log data and sends it as metrics to the Wavefront service.
Setup
This integration uses the Filebeat output plugin
or Wavefront output plugin for Logstash
to collect log data metrics from Logstash.
Step 1. Install the Wavefront Proxy
If you do not have a Wavefront proxy installed on your network, install a proxy.
Step 2. Collect Log Data
You can collect log data using Filebeat or using the Wavefront Output Plugin.
Option 1. Collect Log Data Metrics Using Filebeat
Follow the instructions in Log Data Metrics Integration for configuring the grok patterns to extract metrics from log data and sending data using Filebeat.
Option 2. Collect Log Data Metrics Using the Wavefront Output Plugin for Logstash
- Install Wavefront Logstash Output Plugin
- Install ruby.
- Install ruby bundler –
gem install bundler
. - Clone logstash-output-wavefront and
cd
to the directory. - Build the plugin –
gem build logstash-output-wavefront.gemspec
. - Install the plugin –
logstash-plugin install *wavefront*.gem
.
- Create a config file that specifies wavefront as the output plugin and specifies settings for other plugins. You can see some examples under Optional Configuration below.
output { wavefront { host => "<Proxy-IP>" } }
Optional Configuration
port Metric Port (Default - 2878) prefix Metric Prefix (Default - "logstash") metrics List of metrics (Default - ["count", "mean"]) source Metric source (Default - Hostname of the node running logstash)
You can send log events to Wavefront using the output plugin for Logstash. The events must have the following format:
{ "bytes" => { "count" => 200, "mean" => 42.2 }, "error" => { "count" => 123, "code" => 404 }, "message" => "I'm not a hash type, so I won't get sent." }
The Wavefront output plugin for Logstash generates the following metrics from the event and sends the metrics to Wavefront:
logstash.bytes.count 200 logstash.bytes.mean 42.2 logstash.error.count 123
You can send point tags for a metric to Wavefront using the Wavefront output plugin for Logstash. The event must have the following format:
{ "bytes.tagz.type=access.region=mumbai" => { "count" => 200, "mean" => 42.2 }, "error" => { "count" => 123, "code" => 404 }, "message" => "I'm not a hash type, so I won't get sent." }
Below metrics are the output of the above event:
logstash.bytes.count 200 type=access region=mumbai logstash.bytes.mean 42.2 type=access region=mumbai logstash.error.count 123
Note: In this example the Wavefront output plugin has dropped the
logstash.error.code
metric because the defaultmetrics
list only includescount
andmean
. To include thecode
metric, override the defaultmetrics
to["count", "mean", "code"]
in thewavefront
output plugin. - Start logstash and specify the configuration file with the -f flag.
bin/logstash -f <config-file>
Click here for more information.