Use Wavefront Top or Spy to Investigate Traffic

VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront) includes HTTP spy endpoints for sampling the data that your service instance is currently ingesting. Examining these endpoints helps you understand the data shape and avoid slowdown or other problems.

Get Started with Wavefront Top and Spy

Use spy to extract information programmatically. This page gives details on the available endpoints and associated parameters.

Use Wavefront top if you want a keyboard-driven UI that’s similar to the Linux top UI. The tool is open source and on GitHub. You can also read the blog that Joanna prepared.Note that this video was created in 2019 and some of the information in it might have changed. It also uses the 2019 version of the UI.

You can also watch the video here .

Why Spy?

The spy endpoints can provide insight into new data that is being ingested by your service instance. For example, you might analyze spy results to:

• Verify that your service instance is ingesting the data points that you expect.
• Troubleshoot a sudden change in the rate at which new data is ingested.

Operations for Applications supports the spy endpoints shown in the following table:

Each endpoint displays a header that describes your request, and then lists the results, if any, in close to real time (as soon as they are available). Each returned point, span, or ID is listed on a separate line.

A spy endpoint always returns a sample of the requested data. Operations for Applications always uses multiple data ingesters, but the endpoint connects to a single data ingester. The sample is taken from data that is ingested by that one data ingester. As a result, you always get a sample even if you specify a 100% sample size as an endpoint parameter.

Get Ingested Metric Points with Spy

Your service instance includes an HTTP endpoint that returns a sampling of the ingested metric points that have specified characteristics. You can use the returned list of points to help you answer questions like:

• Show me some ingested points with metric names that start with the prefix Cust.
• How many PPS come from hosts with names that start with the prefix web?
• What are some points that are tagged with env=prod?

Endpoint and Parameters for Metric Points

To get a sampling of ingested data points, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/points

To get a sampling of points with specific characteristics, add one or more of the following parameters:

Example Requests for Metric Points

Suppose you have a service instance named ex1.

Get Ingested Delta Counters with Spy

Your service instance includes an HTTP endpoint that returns a sample of ingested delta counters. The data you see are the aggregated points in each minute bucket that Operations for Applications stores post aggregation, not the individual points that were sent to Operations for Applications.

You can use the returned list to help you answer questions like:

• Show me some ingested delta counters with names that start with the prefix Cust.
• Show me some ingested delta counters with source names that start with the prefix web.
• What are some delta counters that are tagged with env?

Endpoint and Parameters for Delta Counters

To get a sample of ingested delta counters (post aggregation), use the following endpoint. Replace <your_instance> with the name of your service instance:

https://<your_instance>.wavefront.com/api/spy/deltas

To get a sample of delta counters with specific characteristics, add one or more of the following parameters:

Example Requests for Delta Counters

Suppose you have a service instance named ex1.

Get Ingested Histograms with Spy

Your service instance includes an HTTP endpoint that returns a sampling of ingested histograms with specified characteristics.

You can use the returned list of histograms to help you answer questions like this:

• Show me some ingested histograms with names that start with the prefix order.
• How many histograms-per-second come from hosts with names that start with the prefix web?
• What are some histograms that are tagged with cluster or shard?

Endpoint and Parameters for Histograms

To get a sampling of ingested histograms, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/histograms

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

Example Requests for Histograms

Suppose you have a service instance named ex1.

Get Ingested Spans with Spy

Your service instance includes an HTTP endpoint that returns a sampling of ingested spans with specified characteristics. You can use the returned list of spans to help you answer questions like:

• Show me some ingested spans with names that start with the prefix order.
• How many spans-per-second come from hosts with names that start with the prefix web?
• What are some spans that are tagged with cluster or shard?

Endpoint and Parameters for Spans

To get a sampling of ingested spans, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/spans

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

Example Requests for Spans

Suppose you have a service instance named ex1.

Get Ingested Span Logs with Spy

Span logs capture span-specific logging information and are supported by the OpenTracing standard. Some of our SDKs include span logs for errors by default. To get access to other span log information, you can customize an SDK to include span log information or instrument your application.

You can use the returned list of span logs to find out if it contains the data that you expect.
Example:

  traceId="00000000-0000-0000-0000-00000001234" spanId="00000000-0000-0000-0000-000000012341"
  logs=[{"timestamp": 1572303812999, "fields": {"event": "error", "error.kind": "exception"}}]

Endpoint and Parameters for Span Logs

To get a sample of the ingested span logs, use the following endpoint. Replace <your_instance> with the name of your service instance:

https://<your_instance>.wavefront.com/api/gateway/spy/spanlogs

By default, the sampling rate is 1%, which means that Operations for Applications returns 1% of the data. To sample the span logs at a different sampling rate, add the sampling parameter to the URL.

Example Requests for Span Logs

Suppose you have a service instance named ex1 and want to spy on the ingested span logs at a sampling rate of 5%:

http://ex1.wavefront.com/api/gateway/spy/spanlogs?sampling=0.05

Get New ID Assignments with Spy

During ingestion, Operations for Applications assigns an ID to each newly added metric name, span name, source name, and key=value string of a point tag or span tag. A new ID generally indicates that a new time series has been introduced.

Your service instance includes an HTTP endpoint that provides a window into the current stream of new ID assignments. You can use the returned list of ID assignments to see if the data that is currently being ingested has introduced any metrics, sources, spans, or tags that your service instance hasn’t seen yet.

Endpoint and Parameters for New ID Assignments

To get a list of new ID assignments, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/ids

To get ID assignments for a specific type of new item, add one or more of the following parameters:

Example Requests for New IDs

Suppose you have a service instance named ex1.