Fluentd Integration

Fluentd Integration

Fluentd is an open source data collector for a unified logging layer. By setting up this integration, you can send Fluentd metrics into Tanzu Observability by Wavefront.

1. Fluentd: This integration installs and configures Telegraf to send Fluentd metrics into Wavefront. Telegraf is a light-weight server process capable of collecting, processing, aggregating, and sending metrics to a Wavefront proxy.
2. Fluentd on Kubernetes: This explains the configuration of Wavefront Collector for Kubernetes to scrape Fluentd metrics using auto-discovery and annotation based discovery.

In addition to setting up the metrics flow, this integration also installs dashboards:

• Fluentd
• Fluentd on Kubernetes

Here’s a section of a dashboard displaying Fluentd metrics:

To see a list of the metrics for this integration, select the integration from https://github.com/influxdata/telegraf/tree/master/plugins/inputs.

Fluentd Setup

Use the instructions on this page for monitoring:

• Fluentd - Standalone
• Fluentd on Kubernetes

Fluentd

This integration uses the Fluentd input plugin for Telegraf to get the metrics from Fluentd. If you’ve already installed Telegraf on your servers, you can skip to Step 2.

NOTE: Make sure that Telegraf is of version 1.23.0 or later.

Step 1: Install the Telegraf Agent

Log in to your Wavefront instance and follow the instructions in the Setup tab to install Telegraf and a Wavefront proxy in your environment. If a proxy is already running in your environment, you can select that proxy and the Telegraf install command connects with that proxy. Sign up for a free trial to check it out!

Step 2: Ensure the monitor_agent plugin is configured on Fluentd

In the Fluentd configuration file /etc/td-agent/td-agent.conf, ensure that the monitor_agent plugin is configured as in the example below:

  <source>
    @type monitor_agent
    bind 0.0.0.0
    port 24220
  </source>

Step 3: Configure the Fluentd Input Plugin on Telegraf

Create a fluentd.conf file in /etc/telegraf/telegraf.d and enter the Fluentd plugin configuration as in the following example snippet:

# Read metrics exposed by fluentd monitor_agent plugin
[[inputs.fluentd]]
  ## This plugin reads information exposed by fluentd (using /api/plugins.json endpoint).
  ##
  ## Endpoint:
  ## - only one URI is allowed
  ## - https is not supported
  endpoint = "http://example.com:24220/api/plugins.json"

  ## Define which plugins have to be excluded (based on "type" field - e.g. monitor_agent)
  exclude = [
          "monitor_agent",
          "dummy"
  ]

Step 4: Restart Telegraf

Run sudo service telegraf restart to restart your Telegraf agent.

Fluentd on Kubernetes

This integration supports Fluentd deployment as daemonset using standard fluentd-docker images with Prometheus plugin configuration as mentioned below.

1. Make sure that Fluentd is deployed on your Kubernetes cluster. If not deployed already, you can deploy it by using the sample .yaml files as explained below.

2. Add the following match clause in the existing fluent.conf file by using ConfigMaps and save the file. You can find the fluent.conf file under /fluentd/etc/.

<match **>
  @type stdout
  @id out_prometheus
  <buffer>
    flush_thread_count 8
    flush_interval 5s
    chunk_limit_size 2M
    queue_limit_length 32
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

You can override another configuration file using the same ConfigMap, if needed. See the sample fluentd-config-map.yaml file.

1. Mount the same .conf files present in the /fluentd/etc/ directory by using volume mount in your Fluentd deployment .yaml. For more information, see the sample fluentd-daemonset.yaml file.

2. Annotate the Fluentd daemonset to add Prometheus scrape, scheme, port, and path.

kubectl annotate pod <FLUENTD_POD_NAME> prometheus.io/scrape=true prometheus.io/scheme=http prometheus.io/port=24231 prometheus.io/path=/metrics

For more information, see the following sample .yaml files:

• fluentd-config-map.yaml
• fluentd-rbac.yaml
• fluentd-daemonset.yaml

Configure the Wavefront Collector for Kubernetes

You can configure the Wavefront Collector for Kubernetes to scrape Fluentd metrics exposed to Prometheus by using annotation based discovery. To collect the Fluentd metrics automatically, configure the Wavefront Collector for Kubernetes to use auto-discovery.

If you do not have the Wavefront Collector for Kubernetes installed on your Kubernetes cluster, follow the instructions to add it to your cluster by using Helm or performing Manual Installation. You can check the status of Wavefront Collector and Proxy if you are already monitoring the Kubernetes cluster on the Setup tab of the Kubernetes integration.

NOTE: Make sure that auto discovery enableDiscovery: true and annotation based discovery discovery.disable_annotation_discovery: false are enabled in the Wavefront Collector. They should be enabled by default.

Fluentd

Fluentd on Kubernetes