Use HTTP endpoints to get samples data or IDs, or use wftop to examine them with a keyboard-driven UI.

VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront) includes HTTP spy endpoints for sampling the data that your service instance is currently ingesting. Examining these endpoints helps you understand the data shape and avoid slowdown or other problems.

Get Started with Wavefront Top and Spy

Use spy to extract information programmatically. This page gives details on the available endpoints and associated parameters.

Use Wavefront top if you want a keyboard-driven UI that’s similar to the Linux top UI. The tool is open source and on GitHub. You can also read the blog that Joanna prepared.Note that this video was created in 2019 and some of the information in it might have changed. It also uses the 2019 version of the UI.

You can also watch the video here video camera icon.

To set up the Wavefront top utility you need a valid API token. For an Operations for Applications subscription on VMware Cloud services, you must use a VMware Cloud services access token. To get an access token, generate a VMware Cloud services API token associated with your user account and exchange it for the access token. Make sure that when you do this, you deselect the Send empty value check box for the passcode parameter. For more information and instructions, see Use the Operations for Applications REST API.

Why Spy?

The spy endpoints can provide insight into new data that is being ingested by your service instance. For example, you might analyze spy results to:

  • Verify that your service instance is ingesting the data points that you expect.
  • Troubleshoot a sudden change in the rate at which new data is ingested.

Operations for Applications supports the spy endpoints shown in the following table:

Spy EndpointDescription
https://<your_instance>.wavefront.com/api/spy/points Gets new metric data points that are added to existing time series.
https://<your_instance>.wavefront.com/api/spy/deltas Gets new delta counter points that are added to existing time series.
https://<your_instance>.wavefront.com/api/spy/histograms Gets new histograms that are added to existing time series.
https://<your_instance>.wavefront.com/api/spy/spans Gets new spans with existing source names and span tags.
https://<your_instance>.wavefront.com/api/gateway/spy/spanlogs Gets new span logs with the existing trace ID, span ID, and the respective event that created the log.
https://<your_instance>.wavefront.com/api/spy/ids Gets newly allocated IDs that correspond to new metric names, source names, point tags, or span tags. A new ID generally indicates that a new time series has been introduced.

Each endpoint displays a header that describes your request, and then lists the results, if any, in close to real time (as soon as they are available). Each returned point, span, or ID is listed on a separate line.

A spy endpoint always returns a sample of the requested data. Operations for Applications always uses multiple data ingesters, but the endpoint connects to a single data ingester. The sample is taken from data that is ingested by that one data ingester. As a result, you always get a sample even if you specify a 100% sample size as an endpoint parameter.

Get Ingested Metric Points with Spy

Your service instance includes an HTTP endpoint that returns a sampling of the ingested metric points that have specified characteristics. You can use the returned list of points to help you answer questions like:

  • Show me some ingested points with metric names that start with the prefix Cust.
  • How many PPS come from hosts with names that start with the prefix web?
  • What are some points that are tagged with env=prod?

Endpoint and Parameters for Metric Points

To get a sampling of ingested data points, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/points

To get a sampling of points with specific characteristics, add one or more of the following parameters:

ParameterTypeDescription
metric string, optional List a point only if its metric name starts with the specified case-sensitive prefix.
E.g., metric=Cust matches metrics named Customer, Customers, Customer.alerts, but not customer.
host or source string, optional List a point only if its source name starts with the specified case-sensitive prefix.
pointTagKey string, optional List a point only if it has the specified point tag key. Add this parameter multiple times to specify multiple point tags, e.g., pointTagKey=env&pointTagKey=datacenter
sampling decimal, optional 0 to .05, with 0.05 being 5%.
wavefrontSource string, optional Filter for only points that are ingested through a particular proxy or user’s token. To filter for a certain proxy, use this syntax: proxy::<proxy_source>
usage Boolean, optional Flag that determines whether usage information is shown.
usageThresholdDays integer, optional If usage is set to true, specifies the number of days to look back to determine whether data has been accessed; defaults to 7
includeScalingFactor Boolean, optional Flag that determines whether the number of data ingesters is included in the output.

Example Requests for Metric Points

Suppose you have a service instance named ex1.

To List a Sample of
These Points
Use This Query URL
Ingested points for any metric. http://ex1.wavefront.com/api/spy/points
Ingested points with metric names that start with Cust. http://ex1.wavefront.com/api/spy/points?metric=Cust
Ingested points that have point tags named env and loc. http://ex1.wavefront.com/api/spy/points?pointTagKey=env&pointTagKey=loc
Ingested points from a source whose name starts with web1. http://ex1.wavefront.com/api/spy/points?host=web1

Get Ingested Delta Counters with Spy

Your service instance includes an HTTP endpoint that returns a sample of ingested delta counters. The data you see are the aggregated points in each minute bucket that Operations for Applications stores post aggregation, not the individual points that were sent to Operations for Applications.

You can use the returned list to help you answer questions like:

  • Show me some ingested delta counters with names that start with the prefix Cust.
  • Show me some ingested delta counters with source names that start with the prefix web.
  • What are some delta counters that are tagged with env?

Endpoint and Parameters for Delta Counters

To get a sample of ingested delta counters (post aggregation), use the following endpoint. Replace <your_instance> with the name of your service instance:

https://<your_instance>.wavefront.com/api/spy/deltas

To get a sample of delta counters with specific characteristics, add one or more of the following parameters:

ParameterTypeDescription
counter string, optional List a delta counter only if its name starts with the specified case-sensitive prefix
E.g., counter=Cust matches counters named Customer, Customers, Customer.alerts, but not customer.
host string, optional List a delta counter only if its source name starts with the specified case-sensitive prefix.
counterTagKey string, optional List a delta counter only if it has the specified tag key. Add this parameter multiple times to specify multiple point tags, e.g., counterTagKey=env&counterTagKey=datacenter
sampling decimal, optional 0 to .05, with 0.05 being 5%
wavefrontSource string, optional Filter for only points that are ingested through a particular proxy or user’s token. To filter for a certain proxy, use this syntax: proxy::<proxy_source>
usage Boolean, optional Flag that determines whether usage information is shown.
usageThresholdDays integer, optional If usage is set to true, specifies the number of days to look back to determine whether data has been accessed; defaults to 7
includeScalingFactor Boolean, optional Flag that determines whether the number of data ingesters is included in the output.

Example Requests for Delta Counters

Suppose you have a service instance named ex1.

List Sample of
These Delta Counters
With This Query URL
Any ingested delta counters. http://ex1.wavefront.com/api/spy/deltas
Ingested delta counters with names that start with Cust. http://ex1.wavefront.com/api/spy/deltas?counter=Cust
Ingested delta counters that have tags named env and loc. http://ex1.wavefront.com/api/spy/deltas?counterTagKey=env&counterTagKey=loc
Ingested delta counters from a source whose name starts with web1. http://ex1.wavefront.com/api/spy/deltas?host=web1

Get Ingested Histograms with Spy

Your service instance includes an HTTP endpoint that returns a sampling of ingested histograms with specified characteristics.

You can use the returned list of histograms to help you answer questions like this:

  • Show me some ingested histograms with names that start with the prefix order.
  • How many histograms-per-second come from hosts with names that start with the prefix web?
  • What are some histograms that are tagged with cluster or shard?

Endpoint and Parameters for Histograms

To get a sampling of ingested histograms, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/histograms

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

ParameterTypeDescription
histogram string, optional List a histogram only if its name starts with the specified case-sensitive prefix.
E.g., histogram=orderShirt matches histograms named orderShirt and orderShirts, but not OrderShirts.
host string, optional List a histogram only if the name of its source starts with the specified case-sensitive prefix.
histogramTagKey string, optional List a histogram only if it has the specified tag key. Add this parameter multiple times to specify multiple tags, e.g. histogramTagKey=cluster&histogramTagKey=shard
sampling decimal, optional 0 to .05, with 0.05 being 5%
wavefrontSource string, optional Filter for only points that are ingested through a particular proxy or user’s token. To filter for a certain proxy, use this syntax: proxy::<proxy_source>
usage Boolean, optional Flag that determines whether usage information is shown.
usageThresholdDays integer, optional If usage is set to true, specifies the number of days to look back to determine whether data has been accessed; defaults to 7
includeScalingFactor Boolean, optional Flag that determines whether the number of data ingesters is included in the output.

Example Requests for Histograms

Suppose you have a service instance named ex1.

To Get a Sample of
These Histograms
Use This Request
Ingested histograms representing any operation. http://ex1.wavefront.com/api/spy/histograms
Ingested histograms with names that begin with orderShirts. http://ex1.wavefront.com/api/spy/histograms?histogram=orderShirts
Ingested histograms that have the specified sampling rate, tag key `cluster`, and histogram name `orderShirt`. http://ex1.wavefront.com/api/spy/histograms?sampling=0.004&histogramTagKey=cluster&histogram=orderShirt
Ingested histograms from a host whose name starts with web1. http://ex1.wavefront.com/api/spy/histograms?host=web1

Get Ingested Spans with Spy

Your service instance includes an HTTP endpoint that returns a sampling of ingested spans with specified characteristics. You can use the returned list of spans to help you answer questions like:

  • Show me some ingested spans with names that start with the prefix order.
  • How many spans-per-second come from hosts with names that start with the prefix web?
  • What are some spans that are tagged with cluster or shard?

Endpoint and Parameters for Spans

To get a sampling of ingested spans, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/spans

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

ParameterTypeDescription
name string, optional List a span only if its operation name starts with the specified case-sensitive prefix.
E.g., name=orderShirt matches spans named orderShirt and orderShirts, but not OrderShirts.
hostPrefix string, optional List a span only if the name of its source starts with the specified case-sensitive prefix.
spanTagKey string, optional List a span only if it has the specified span tag key. Add this parameter multiple times to specify multiple span tags, e.g. spanTagKey=cluster&spanTagKey=shard
sampling decimal, optional 0 to .05, with 0.05 being 5%.

Example Requests for Spans

Suppose you have a service instance named ex1.

To Get a Sample of
These Spans
Use This Request
Ingested spans representing any operation. http://ex1.wavefront.com/api/spy/spans
Ingested spans with names that begin with orderShirts. http://ex1.wavefront.com/api/spy/spans?name=orderShirts
Ingested spans that have span tags cluster and shard. http://ex1.wavefront.com/api/spy/spans?spanTagKey=cluster&spanTagKey=shard
Ingested spans from a host whose name starts with web1. http://ex1.wavefront.com/api/spy/spans?host=web1

Get Ingested Span Logs with Spy

Span logs capture span-specific logging information and are supported by the OpenTracing standard. Some of our SDKs include span logs for errors by default. To get access to other span log information, you can customize an SDK to include span log information or instrument your application.

You can use the returned list of span logs to find out if it contains the data that you expect.
Example:

  traceId="00000000-0000-0000-0000-00000001234" spanId="00000000-0000-0000-0000-000000012341"
  logs=[{"timestamp": 1572303812999, "fields": {"event": "error", "error.kind": "exception"}}]

Endpoint and Parameters for Span Logs

To get a sample of the ingested span logs, use the following endpoint. Replace <your_instance> with the name of your service instance:

https://<your_instance>.wavefront.com/api/gateway/spy/spanlogs

By default, the sampling rate is 1%, which means that Operations for Applications returns 1% of the data. To sample the span logs at a different sampling rate, add the sampling parameter to the URL.

ParameterTypeDescription
sampling decimal, optional 0 to .05, with 0.05 being 5%.

Example Requests for Span Logs

Suppose you have a service instance named ex1 and want to spy on the ingested span logs at a sampling rate of 5%:

http://ex1.wavefront.com/api/gateway/spy/spanlogs?sampling=0.05

Get New ID Assignments with Spy

During ingestion, Operations for Applications assigns an ID to each newly added metric name, span name, source name, and key=value string of a point tag or span tag. A new ID generally indicates that a new time series has been introduced.

Your service instance includes an HTTP endpoint that provides a window into the current stream of new ID assignments. You can use the returned list of ID assignments to see if the data that is currently being ingested has introduced any metrics, sources, spans, or tags that your service instance hasn’t seen yet.

Endpoint and Parameters for New ID Assignments

To get a list of new ID assignments, use the following endpoint. Replace <your_instance> with the name of your service instance:

  https://<your_instance>.wavefront.com/api/spy/ids

To get ID assignments for a specific type of new item, add one or more of the following parameters:

ParameterTypeDescription
name string, optional Case-sensitive prefix for the items that you are interested in.
type string, optional Type of new items you want to see ID assignments for:
  • METRIC - Metric names
  • SPAN - Span names
  • HOST - Source names
  • STRING - Point tags or span tags, represented as a single string containing a unique key-value pair, e.g. env=prod, env=dev, etc.
sampling decimal, optional 0 to .05, with 0.05 being 5%
includeScalingFactor Boolean, optional Flag that determines whether the number of data ingesters is included in the output.

Example Requests for New IDs

Suppose you have a service instance named ex1.

To Get ID Assignments
For These Items
Use This Request
All metric names, span names, source names, and tags. http://ex1.wavefront.com/api/spy/ids
All metric names. http://ex1.wavefront.com/api/spy/ids?type=METRIC
Metric names that start with cpu. http://ex1.wavefront.com/api/spy/ids?type=METRIC&name=cpu
Key-value pairs with keys that start with comp. http://ex1.wavefront.com/api/spy/ids?type=STRING&name=comp
Key-value pairs of the form loc=palo. http://ex1.wavefront.com/api/spy/ids?type=STRING&name=loc%3Dpalo
Source names that start with web1. http://ex1.wavefront.com/api/spy/ids?type=HOST&name=web1