Set up the Tanzu Observability tile and monitor your environment.

VMware Tanzu Application Service, previously known as Pivotal Cloud Foundry, is a popular platform for building cloud-native applications.

Tanzu Observability by Wavefront (Wavefront) is a cloud-hosted service for full-featured observability. When you Tanzu Application Service to send data to the Wavefront proxy, you can take advantage of preconfigured dashboards, clone and customize dashboards, and more.

Product Snapshot

Version, Release Date v4.1, May 2022
Compatible Ops Manager versions v2.10 and later
Compatible Tanzu Application Service versions v2.11 and later
IaaS support AWS, Azure, GCP, OpenStack, and vSphere

Videos

Watch these videos to get started:

TAS Monitoring and Alerting with Tanzu Observability



This 60-second video gives an overview of the architecture, installation process, and benefits of the Tanzu Observability integration for Tanzu Application Service.

You can also watch the video in the Tanzu Observability Integrations playlist here video camera icon.

Integrating TAS with Tanzu Observability: How to Get Started



This video is a complete walk-through of the steps required to get your data flowing from Tanzu Application Service to Tanzu Observability. You learn how to set up the nozzle in Ops Manager, how to get data flowing, and how you can view your data in preconfigured Tanzu Observability dashboards.

You can also watch the video in the Tanzu Observability Integrations playlist here video camera icon.

Process Overview

We’ve streamlined the getting started process so it involves a few simple steps – some are performed in Tanzu Ops Manager, and some in the Wavefront GUI.

4 steps below shown in an image. First 2 are purple, pivotal, next 2 are blue, wavefront.

  1. Download the Tanzu Observability by Wavefront nozzle file from the Tanzu Network
  2. In Tanzu Ops Manager, install, configure, and deploy the nozzle. At a minimum:
    1. In the Wavefront Proxy Config section, specify the Wavefront instance and API token (shown in the Wavefront instance in the integration’s Setup tab) and a user-friendly host name. OpsMan Proxy Config tab with the 3 required items highlighted
    2. In the Telegraf Agent Config section, specify the Foundation name.

      See Ops Manager: Install, Configure, and Deploy the Nozzle and Tanzu Application Service to Tanzu Observability FAQs After you complete nozzle deployment, metrics are flowing from Tanzu Application Service to the Wavefront proxy and from there to your Wavefront instance. See Data Flow below.
  3. Log in to your Wavefront instance (for example, https://example.wavefront.com) and confirm that metrics are flowing:
    1. Click Integrations in the toolbar, search for Tanzu Application Service, and select the integration.
    2. Click the Metrics tab and confirm metrics are flowing.
  4. With the integration selected, click the Dashboards tab.
    1. Select from the set of predefined dashboards, which are modeled on the corresponding Healthwatch dashboards but have additional options. For example, you can examine multiple foundations from one dashboard.
    2. Explore one or two dashboards. Examine Data with Dashboards and Charts has an overview and includes a video.
    3. As appropriate, clone any of the existing dashboards to add charts, modify queries, and more. See Create, Customize, and Optimize Dashboards and Create and Customize Charts

Architecture and Data Flow

Here’s an overview of the flow of data from the Tanzu Application Service Firehose through the nozzle to the Wavefront service. The nozzle consists of these main components:

  • Healthwatch Exporters: Exporters are deployed as VMs.
  • Telegraf: An open source, lightweight server process for collecting, processing, and aggregating metrics.
  • Wavefront Proxy: Ingests metrics and forwards them to the Wavefront service in a secure, fast, and reliable manner

Here’s the data pipeline:

  1. The Healthwatch Exporter VMs stream metrics from the Tanzu Application Service Firehose. The tile creates a VM for each Healthwatch exporter, a VM for Telegraf, and a VM for the Wavefront proxy.
  2. Telegraf scrapes the VMs at a predefined interval, and converts them to Wavefront data format. Telegraf uses a built-in plugin. It uses the Wavefront golang SDK to convert the data.
  3. Next, Telegraf sends the data to the Wavefront proxy.
  4. The proxy send the metrics to the Wavefront service.

TAS Firehose to Exporters like pas-sli-exporter, to Telegraf agent, to Wavefront proxy, to Wavefront service

Tanzu Application Service to Tanzu Observability FAQs

In this section, we have some answers to frequently asked questions.

How can I change the scrape interval?

By default, the scrape interval is set to 15 seconds, but you can set up your environment to check more frequently:

  1. In OpsManager, click Telegraf Agent Config.
  2. Set the Scrape Interval (seconds) field and click Save.

How can I send TAS data via a proxy that is deployed outside the tile?

Some customers have a central monitoring/observability team that requires that all data to be sent via a specific set of production proxies. Those proxies are used to filter or alter data before they are sent to the Wavefront service.

You can set up your environment to use production proxies as follows:

  1. In Ops Manager, click Telegraf Agent Config.
  2. In Advanced Options, select Yes.
  3. In the Custom Proxy URL field, provide a proxy URL or IP in the Custom Proxy URL. This could also be load balancer URL if the external proxies are behind a load balancer.
  4. (Optional) In the Proxy Port field, provide a custom proxy port (Default is 2878).
  5. Click Save.

How can I customize metrics ingestion?

If you don’t want to monitor some of your TAS platform metrics, you can choose not to send them. If those metrics are monitored with any out-of-the-box or custom dashboards, they show up as No Data.

For example, if you don’t want to ingest certificate expiration metrics, then you can remove the VM instance that is assigned to the Cert Expiration Exporter by default. All the metrics that this exporter scrapes will not get ingested.

  1. In Ops Manager, click Resource Config.
  2. Find the exporter for which you don’t want to emit metrics, set it to 0, and click Save. The screenshot below shows how to do this.

Cert Expiration Exporter is in process of being changed from Automatic to 0

How can I customize proxy behavior?

The Wavefront proxy allows you to control many aspects of your ingestion pipeline with configuration properties and preprocessor rules.

  • Configuration file: The proxy processes data according to a configuration file. You can modify configuration properties – for example, to create block list and allow list regex patterns, specify information about certain data formats, and much more. See Configuring Wavefront Proxies.
  • Preprocessor Rules: Starting with proxy version 4.1, the Wavefront proxy includes a preprocessor that applies user-defined rules before data is sent to the Wavefront service. You can use preprocessor rules to correct certain data quality issues when you can’t fix the problem at the emitting source. See Configuring Wavefront Proxy Preprocessor Rules.

You can specify custom elements as follows:

  1. In OpsManager, click Wavefront Proxy Config.
  2. Click Wavefront Proxy Config, and then click Custom.
  3. Make your changes and click Save.

Things aren’t working. What can I do?

Ensure that your environment meet requirements on the Ops Manager side and on the Tanzu Observability side.

  • Ops Manager Requirements VMware Tanzu Observability by Wavefront nozzle has the following requirements:
    • Read-only access to the Doppler Firehose and Cloud Controller.
    • Access to a Wavefront instance and an API token. Service Account API token is recommended.
  • Tanzu Observability by Wavefront Requirements To set up the Tanzu Application Service integration on you Wavefront instance, you must have:
    • Access to a Wavefront instance with a URL like https://example.wavefront.com.
    • At a minimum, Integrations permission on that Wavefront instance.
    • This version of the Tanzu Observability by Wavefront nozzle is compatible with Wavefront proxy version 10.14 and later.

See Tanzu Observability and TAS Troubleshooting for more.

Process Details

This section explains each step in the flow in detail.

Prerequisites

  • You must have a Tanzu Network account to configure the Tanzu Observability Nozzle in Ops Manager.
  • You must have login credentials for a Wavefront instance to configure the integration.
  • Ensure that the root certificate for Ops Manager is included in data coming from Tanzu Application Service. See the No Data Flowing and Certificate Error troubleshooting section.

Step 1: Download the Nozzle File

Download the Tanzu Observability by Wavefront nozzle file from the Tanzu Network to your local filesystem. You must have a Tanzu Network account to perform this task.

Step 2: Ops Manager: Install, Configure, and Deploy the Nozzle

To install the nozzle:

  1. Download the VMware Tanzu Observability TAS tile version 4 from VMware Tanzu Network.
  2. Log in to Ops Manager, select Installation Dashboard, click Import a Product, and upload the file you just downloaded.
  3. Under Import a Product, click the plus sign (+) next to the version number of VMware Tanzu Observability by Wavefront nozzle. This adds the tile to your staging area.

The tile is now available, but the orange bar at the bottom indicates that the product is not yet configured.

To configure the nozzle:

In Ops Manager, click the Tanzu Observability by Wavefront tile. With Settings selected (the default), follow these steps:

Step 1. Click Assign AZs and Networks, to configure the availability zone and network settings.
  1. Under Place singleton jobs in, select the AZ you want to use. Ops Manager runs any job with a single instance in this AZ.
  2. Under Balance other jobs in, select one or more other AZs. Ops Manager balances instances of jobs with more than one instance across the AZs that you specify.
  3. From the Network drop-down menu, select the subnet that you created when you configured the BOSH Director tile. That network often has pas or tas in its name.
  4. From the Service Network drop-down menu, select the service subnet that you created when you configured the BOSH Director tile. That network often has services in its name.
  5. Click Save.
Assign AZ and Networks screenshot, with values as discussed in text above.
Step 2. Click Wavefront Proxy Config and specify:
  1. The URL of your Wavefront instance, for example, https://example.wavefront.com.
  2. A Wavefront API token. See Generating an API Token.
  3. User-friendly name for the proxy.
  4. Click Save or click Custom (see the next step). The nozzle ignores these configuration properties because they're already defined in the setup steps:
    • server
    • hostname
    • pushListenerPorts
    • opentsdbPorts
    • idFile
    • buffer
    • preprocessorConfigFile
  5. (TAS Nozzle v3) If you are currently using the Tanzu Application Service v3 service broker to send metrics from apps to the Wavefront proxy, select this check box to ensure existing bindings continue to work. Otherwise, leave the check box deselected.
  6. (Optional) If you click Custom you can specify proxy configuration properties or proxy preprocessor rules.
    • In the Config field, specify one or more configuration properties and values, separated by newline characters. For example pushRateLimit=10000.
    • In the Preprocessor Rules field, specify one or more preprocessor rules, separated by newline characters. For example: '2878': - rule : example-replace-badchars action : replaceRegex scope : pointLine search : "[&\\$\\*]" replace : "_" - rule : drop-az-tag action : dropTag tag : az match : dev.*.
  7. Click Save.
Proxy Config screenshot, with values as discussed in text above.
Step 3. Click Telegraf Agent Config and customize the Telegraf Agent config or accept the defaults.
  1. For Scrape Interval (seconds), specify the default interval at which Telegraf agent checks for new data.
  2. For Metric Buffer Limit, specify the size of the buffer that Telegraf uses to queue data. If your environment sends bursty data, use a larger buffer.
  3. For Flush Interval (seconds), specify how often data are flushed from each output plugin to the Wavefront proxy. Must be lower than the scrape interval.
  4. For Foundation Name, specify a unique name for your Tanzu Application Service environment. This name will be added to all metrics as the metrics source (source=).
  5. (Optional) Click Advanced Options to specify a custom proxy URL, custom proxy port, or additional Telegraf inputs.
  6. Click Save.
Telegraf Agent Config screenshot, with values as discussed in text above.
Step 4. Click Metrics Exporters to customize metrics export from Tanzu Application Service.
  1. Select Skip TLS Verification When Querying if you want to turn off TLS verification, for example, during testing or a POC.
  2. Select a BOSH Health Check Availability Zone if you don't want to use the default zone.
  3. Optionally, select the BOSH Health Check Payload VM Type and change the default. In almost all cases users don't change this field.
  4. To collect metrics of all the apps bound to a service instance, select the Retrieve Service Binding Metrics check box.
    This way you can, for example, find the Redis instance or MySQL database that is bound to a particular app. If you turn this setting on, you might experience performance issues, because the information is retrieved by making API calls to the CAPI component within Tanzu Application Service. In some environments that might add too much load on the CAPI servers.
  5. Click Save.
Metric Exporter screenshot, with values as discussed in text to the left.
(Optional) Step 5. Errands are scripts that can run at the beginning and at the end of an installed product’s availability time. Most users don't make changes to Errands for this nozzle. Errands is selected, and defaults are show.
(Optional) Step 6. Click Resource Config to review the VM sizing for the deployment. You can choose smaller than default VMs to save money on small and noncritical foundations, and very large VMs with lots of CPU and MEM to scale for large foundations with high volumes of metrics.
Note: SM Forwarder is set to Automatic:0. Do not change this setting.
Resource Config is selected and defaults of first 4 items are shown

Step 3: Check That Metrics are Flowing and Examine Your Data

Log in to your Wavefront instance (for example, https://example.wavefront.com) and confirm that metrics are flowing:

  1. Click Integrations in the toolbar, search for Tanzu Application Service, and select the integration.
  2. Click the Metrics tab and confirm metrics are flowing.

Step 4: Use Dashboard to Examine Your Data

  1. With the integration selected, click the Dashboards tab.
  2. Select from the set of predefined dashboards, which are modeled on the corresponding Healthwatch dashboards but have additional options. For example, you can examine multiple foundations from one dashboard.
  3. Explore one or two dashboards. Examine Data with Dashboards and Charts has an overview and includes a video.
  4. As appropriate, clone any of the existing dashboards to add charts, modify queries, and more. See Create, Customize, and Optimize Dashboards and Create and Customize Charts.

Learn More!