Use HTTP endpoints to get samples data or IDs, or use wftop to examine them with a keyboard-driven UI.

Your Wavefront instance includes HTTP spy endpoints for sampling the data that your Wavefront instance is currently ingesting. Examining these endpoints helps you understand the data shape and avoid slowdown or other problems.

Get Started with Wavefront Top and Spy

Use spy to extract information programmatically. This page gives details on the available endpoints and associated parameters.

Use Wavefront top if you want a keyboard-driven UI that’s similar to the Linux top UI. The tool is open source and on Github. You can also read the blog that Joanna prepared, or watch a short video.

Why Spy?

The Wavefront spy endpoints can provide insight into new data that is being ingested by your Wavefront instance. For example, you might analyze spy results to:

  • Verify that your Wavefront instance is ingesting the data points that you expect.
  • Troubleshoot a sudden change in the rate at which new data is ingested.

Wavefront supports the spy endpoints shown in the following table:

Spy EndpointDescription
https://<cluster>.wavefront.com/api/spy/points Gets new metric data points that are added to existing time series.
https://<cluster>.wavefront.com/api/spy/histograms Gets new histograms that are added to existing time series.
https://<cluster>.wavefront.com/api/spy/spans Gets new spans with existing source names and span tags.
https://<cluster>.wavefront.com/api/gateway/spy/spanlogs Gets new span logs with the existing trace ID, span ID, and the respective event that created the log.
https://<cluster>.wavefront.com/api/spy/ids Gets newly allocated IDs that correspond to new metric names, source names, point tags, or span tags. A new ID generally indicates that a new time series has been introduced.

Each endpoint displays a header that describes your request, and then lists the results, if any, in close to real time (as soon as they are available). Each returned point, span, or ID is listed on a separate line.

Get Ingested Metric Points with Spy

Your Wavefront instance includes an HTTP endpoint that returns a sampling of the ingested metric points that have specified characteristics. You can use the returned list of points to help you answer questions like:

  • Show me some ingested points with metric names that start with the prefix Cust.
  • How many pps come from hosts with names that start with the prefix web?
  • What are some points that are tagged with env=prod?

Endpoint and Parameters for Metric Points

To get a sampling of ingested data points, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

  https://<cluster>.wavefront.com/api/spy/points

To get a sampling of points with specific characteristics, add one or more of the following parameters:

ParameterDescription
metric List a point only if its metric name starts with the specified case-sensitive prefix.
E.g., metric=Cust matches metrics named Customer, Customers, Customer.alerts, but not customer.
host List a point only if its source name starts with the specified case-sensitive prefix.
pointTagKey List a point only if it has the specified point tag key. Add this parameter multiple times to specify multiple point tags, e.g., pointTagKey=env&pointTagKey=datacenter
sampling 0 to 1, with 0.01 being 1%.

Example Requests for Metric Points

Suppose you have a Wavefront instance named ex1.

To List a Sample of
These Points
Use This Query URL
Ingested points for any metric. http://ex1.wavefront.com/api/spy/points
Ingested points with metric names that start with Cust. http://ex1.wavefront.com/api/spy/points?metric=Cust
Ingested points that have point tags named env and loc. http://ex1.wavefront.com/api/spy/points?pointTagKey=env&pointTagKey=loc
Ingested points from a source whose name starts with web1. http://ex1.wavefront.com/api/spy/points?host=web1

Get Ingested Histograms with Spy

Your Wavefront instance includes an HTTP endpoint that returns a sampling of ingested histograms with specified characteristics. Sampling rate here is a display sampling rate (1% by default). For example, if you set the rate to 0.3, then spy shows 30% of results. By default, sampling rate is 1%, which means we return 1% of the data.

You can use the returned list of histograms to help you answer questions like this:

  • Show me some ingested histograms with names that start with the prefix order.
  • How many histograms-per-second come from hosts with names that start with the prefix web?
  • What are some histograms that are tagged with cluster or shard?

Endpoint and Parameters for Histograms

To get a sampling of ingested histograms, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

  https://<cluster>.wavefront.com/api/spy/histograms

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

ParameterDescription
histogram List a histogram only if its name starts with the specified case-sensitive prefix.
E.g., histogram=orderShirt matches histograms named orderShirt and orderShirts, but not OrderShirts.
host List a histogram only if the name of its source starts with the specified case-sensitive prefix.
histogramTagKey List a histogram only if it has the specified tag key. Add this parameter multiple times to specify multiple tags, e.g. histogramTagKey=cluster&histogramTagKey=shard
sampling 0 to 1, with 0.01 being 1% (the default). Sampling rate affects the display. For example, if you set the rate to 30, then spy only shows 30% of results.

Example Requests for Histograms

Suppose you have a Wavefront instance named ex1.

To Get a Sample of
These Histograms
Use This Request
Ingested histograms representing any operation. http://ex1.wavefront.com/api/spy/histograms
Ingested histograms with names that begin with orderShirts. http://ex1.wavefront.com/api/spy/histograms?histogram=orderShirts
Ingested histograms that have the specified sampling rate, tag key `cluster`, and histogram name `orderShirt`. http://ex1.wavefront.com/api/spy/histograms?sampling=0.004&histogramTagKey=cluster&histogram=orderShirt
Ingested histograms from a host whose name starts with web1. http://ex1.wavefront.com/api/spy/histograms?host=web1

Get Ingested Spans with Spy

Your Wavefront instance includes an HTTP endpoint that returns a sampling of ingested spans with specified characteristics. You can use the returned list of spans to help you answer questions like:

  • Show me some ingested spans with names that start with the prefix order.
  • How many spans-per-second come from hosts with names that start with the prefix web?
  • What are some spans that are tagged with cluster or shard?

Endpoint and Parameters for Spans

To get a sampling of ingested spans, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

  https://<cluster>.wavefront.com/api/spy/spans

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

ParameterDescription
name List a span only if its operation name starts with the specified case-sensitive prefix.
E.g., name=orderShirt matches spans named orderShirt and orderShirts, but not OrderShirts.
host List a span only if the name of its source starts with the specified case-sensitive prefix.
spanTagKey List a span only if it has the specified span tag key. Add this parameter multiple times to specify multiple span tags, e.g. spanTagKey=cluster&spanTagKey=shard
sampling 0 to 1, with 0.01 being 1%.

Example Requests for Spans

Suppose you have a Wavefront instance named ex1.

To Get a Sample of
These Spans
Use This Request
Ingested spans representing any operation. http://ex1.wavefront.com/api/spy/spans
Ingested spans with names that begin with orderShirts. http://ex1.wavefront.com/api/spy/spans?name=orderShirts
Ingested spans that have span tags cluster and shard. http://ex1.wavefront.com/api/spy/spans?spanTagKey=cluster&spanTagKey=shard
Ingested spans from a host whose name starts with web1. http://ex1.wavefront.com/api/spy/spans?host=web1

Get Ingested Span Logs with Spy

Span logs capture span-specific logging information and are supported by the OpenTracing standard. Some Wavefront SDKs include span logs for errors by default. To get access to other span log information, you can customize a Wavefront SDK to include span log information or instrument your application.

You can use the returned list of span logs to find out if it contains the data that you expect.
Example:

  traceId="00000000-0000-0000-0000-00000001234" spanId="00000000-0000-0000-0000-000000012341"
  logs=[{"timestamp": 1572303812999, "fields": {"event": "error", "error.kind": "exception"}}]

Endpoint and Parameters for Span Logs

To get a sample of the ingested span logs, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

https://<cluster>.wavefront.com/api/gateway/spy/spanlogs

By default, the sampling rate is 1%, which means Wavefront returns 1% of the data. To sample the span logs at a different sampling rate, add the sampling parameter to the URL.

ParameterDescription
sampling 0 to 1, with 0.01 being 1%.

Example Requests for Span Logs

Suppose you have a Wavefront instance named ex1 and want to spy on the ingested span logs at a sampling rate of 5%:

http://ex1.wavefront.com/api/gateway/spy/spanlogs?sampling=0.05

Get New ID Assignments with Spy

During ingestion, Wavefront assigns an ID to each newly added metric name, span name, source name, and key=value string of a point tag or span tag. A new ID generally indicates that a new time series has been introduced.

Your Wavefront instance includes an HTTP endpoint that provides a window into the current stream of new ID assignments. You can use the returned list of ID assignments to see if the data that is currently being ingested has introduced any metrics, sources, spans, or tags that your Wavefront system hasn’t seen yet.

Endpoint and Parameters for New ID Assignments

To get a list of new ID assignments, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

  https://<cluster>.wavefront.com/api/spy/ids

To get ID assignments for a specific type of new item, add one or more of the following parameters:

ParameterDescription
type Type of new items you want to see ID assignments for:
  • METRIC - Metric names
  • SPAN - Span names
  • HOST - Source names
  • STRING - Point tags or span tags, represented as a single string containing a unique key-value pair, e.g. env=prod, env=dev, etc.
name Case-sensitive prefix for the items that you are interested in.
sampling 0 to 1, with 0.01 being 1%

Example Requests for New IDs

Suppose you have a Wavefront instance named ex1.

To Get ID Assignments
For These Items
Use This Request
All metric names, span names, source names, and tags. http://ex1.wavefront.com/api/spy/ids
All metric names. http://ex1.wavefront.com/api/spy/ids?type=METRIC
Metric names that start with cpu. http://ex1.wavefront.com/api/spy/ids?type=METRIC&name=cpu
Key-value pairs with keys that start with comp. http://ex1.wavefront.com/api/spy/ids?type=STRING&name=comp
Key-value pairs of the form loc=palo. http://ex1.wavefront.com/api/spy/ids?type=STRING&name=loc%3Dpalo
Source names that start with web1. http://ex1.wavefront.com/api/spy/ids?type=HOST&name=web1