Use HTTP endpoints to get samples of the metric points, spans, or ID allocations from your Wavefront instance.

Your Wavefront instance includes HTTP spy endpoints for sampling the data that your Wavefront instance is currently ingesting.

Note: For an interactive alternative to the spy endpoints, use Wavefront Top (wftop), which is an open source tool.

Why spy?

The Wavefront spy endpoints can provide insight into new data that is being ingested by your Wavefront instance. For example, you might analyze spy results to:

  • Verify that your Wavefront instance is ingesting the data points that you expect.
  • Troubleshoot a sudden change in the rate at which new data is ingested.

Wavefront supports the spy endpoints shown in the following table:

Spy EndpointDescription
https://<cluster>.wavefront.com/api/spy/points Gets new metric data points that are added to existing time series.
https://<cluster>.wavefront.com/api/spy/spans Gets new spans with existing source names and span tags.
https://<cluster>.wavefront.com/api/spy/ids Gets newly allocated IDs that correspond to new metric names, source names, point tags, or span tags. A new ID generally indicates that a new time series has been introduced.

Each endpoint displays a header that describes your request, and then lists the results, if any, in close to real time (as soon as they are available). Each returned point, span, or ID is listed on a separate line.

Note: A spy endpoint returns a sample of the requested data, and you specify the sample size as an endpoint parameter. Because the endpoint connects to a single Wavefront back-end, the sample is taken from just the data that is ingested on a single shard, even when you request 100% sampling.

Getting Ingested Metric Points

Your Wavefront instance includes an HTTP endpoint that returns a sampling of the ingested metric points that have specified characteristics. You can use the returned list of points to help you answer questions like:

  • Show me some ingested points with metric names that start with the prefix Cust.
  • How many pps come from hosts with names that start with the prefix web?
  • What are some points that are tagged with env=prod?

Endpoint and Parameters for Metric Points

To get a sampling of ingested data points, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

https://<cluster>.wavefront.com/api/spy/points

To get a sampling of points with specific characteristics, add one or more of the following parameters:

ParameterDescription
metric List a point only if its metric name starts with the specified case-sensitive prefix.
E.g., metric=Cust matches metrics named Customer, Customers, Customer.alerts, but not customer.
host List a point only if its source name starts with the specified case-sensitive prefix.
pointTagKey List a point only if it has the specified point tag key. Add this parameter multiple times to specify multiple point tags, e.g., pointTagKey=env&pointTagKey=datacenter
sampling 0 to 1, with 0.01 being 1%.

Example Requests for Metric Points

Suppose you have a Wavefront instance named ex1.

To List a Sample of
These Points
Use This Query URL
Ingested points for any metric. http://ex1.wavefront.com/api/spy/points
Ingested points with metric names that start with Cust. http://ex1.wavefront.com/api/spy/points?metric=Cust
Ingested points that have point tags named env and loc. http://ex1.wavefront.com/api/spy/points?pointTagKey=env&pointTagKey=loc
Ingested points from a source whose name starts with web1. http://ex1.wavefront.com/api/spy/points?host=web1

Getting Ingested Spans

Your Wavefront instance includes an HTTP endpoint that returns a sampling of ingested spans with specified characteristics. You can use the returned list of spans to help you answer questions like:

  • Show me some ingested spans with names that start with the prefix order.
  • How many spans-per-second come from hosts with names that start with the prefix web?
  • What are some spans that are tagged with cluster or shard?

Endpoint and Parameters for Spans

To get a sampling of ingested spans, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

https://<cluster>.wavefront.com/api/spy/spans

To get a sampling of spans with specific characteristics, add one or more of the following parameters:

ParameterDescription
name List a span only if its operation name starts with the specified case-sensitive prefix.
E.g., name=orderShirt matches spans named orderShirt and orderShirts, but not OrderShirts.
host List a span only if the name of its source starts with the specified case-sensitive prefix.
spanTagKey List a span only if it has the specified span tag key. Add this parameter multiple times to specify multiple span tags, e.g. spanTagKey=cluster&spanTagKey=shard
sampling 0 to 1, with 0.01 being 1%.

Example Requests for Spans

Suppose you have a Wavefront instance named ex1.

To Get a Sample of
These Spans
Use This Request
Ingested spans representing any operation. http://ex1.wavefront.com/api/spy/spans
Ingested spans with names that begin with orderShirts. http://ex1.wavefront.com/api/spy/spans?name=orderShirts
Ingested spans that have span tags cluster and shard. http://ex1.wavefront.com/api/spy/spans?spanTagKey=cluster&spanTagKey=shard
Ingested spans from a host whose name starts with web1. http://ex1.wavefront.com/api/spy/spans?host=web1

Getting New ID Assignments

During ingestion, Wavefront assigns an ID to each newly added metric name, span name, source name, and key=value string of a point tag or span tag. A new ID generally indicates that a new time series has been introduced.

Your Wavefront instance includes an HTTP endpoint that provides a window into the current stream of new ID assignments. You can use the returned list of ID assignments to see if the data that is currently being ingested has introduced any metrics, sources, spans, or tags that your Wavefront system hasn’t seen yet.

Endpoint and Parameters for New ID Assignments

To get a list of new ID assignments, use the following endpoint. Replace <cluster> with the name of your Wavefront instance:

https://<cluster>.wavefront.com/api/spy/ids

To get ID assignments for a specific type of new item, add one or more of the following parameters:

ParameterDescription
type Type of new items you want to see ID assignments for:
  • METRIC - Metric names
  • SPAN - Span names
  • HOST - Source names
  • STRING - Point tags or span tags, represented as a single string containing a unique key-value pair, e.g. env=prod, env=dev, etc.
name Case-sensitive prefix for the items that you are interested in.
sampling 0 to 1, with 0.01 being 1%

Example Requests for New IDs

Suppose you have a Wavefront instance named ex1.

To Get ID Assignments
For These Items
Use This Request
All metric names, span names, source names, and tags. http://ex1.wavefront.com/api/spy/ids
All metric names. http://ex1.wavefront.com/api/spy/ids?type=METRIC
Metric names that start with cpu. http://ex1.wavefront.com/api/spy/ids?type=METRIC&name=cpu
Key-value pairs with keys that start with comp. http://ex1.wavefront.com/api/spy/ids?type=STRING&name=comp
Key-value pairs of the form loc=palo. http://ex1.wavefront.com/api/spy/ids?type=STRING&name=loc%3Dpalo
Source names that start with web1. http://ex1.wavefront.com/api/spy/ids?type=HOST&name=web1