Learn how to query for Wavefront trace data.

After your application sends trace data to Wavefront, you can examine that data in the Traces browser. By fine-tuning the trace query in the Traces browser, you find the traces that you’re interested in by describing the spans they must contain.

Get Started With Trace Queries

You query for traces by using Query Builder to describe the spans they must contain.

  1. Select Applications > Traces in the task bar to display the Traces browser.
  2. Select an operation to specify the scans to be matched. tracing query builder
  3. Optionally add one or more filters to refine your query. tracing query builder
  4. Click Search in the query bar to update the list of traces.

Note: As an alternative, you can use Query Editor to submit advanced trace queries explicitly.

Select an Operation

You select an operation to display traces with at least one span that represents the work done by that operation.

  1. Display the Traces browser.
  2. Click the Operation selector to open the cascading menu.
    tracing query builder operation selector

  3. Select name components for an application, service, and operation.
    tracing query builder operation menu

  4. Either add filters or click Search.

Note: You can select * instead of name components to select all operations in a service, or all operations in all services: tracing query builder operation menu

Note: The names in the cascading menus correspond to tag values and span names that a developer specified while instrumenting the application code.

Add Filters

After you have selected an operation, you can optionally add filters to further describe the traces you want to see. You can add different types of filters in any order.

  1. Select an operation if you have not already done so.
  2. Click Add Filter and select a filter type. tracing query builder filter type menu

  3. Click next to the filter type and specify the filter setting. Depending on the filter type:
    • Select settings from menus.
    • Type one or more settings in fields.
  4. Click Add Filter to add another filter, or click Search.
Choose This Filter TypeTo Return
Tags Traces with at least one span that is associated with a selected tag, or that was emitted from a selected source (host). Use this type for indexed tags that your application uses, typically cluster, shard, component, source, and so on.
RawTags Traces with at least one span that is associated with a specified tag, for example, env="prod". Use this type for custom or unindexed span tags.
Duration Traces that have a specified minimum and/or maximum length (in milliseconds).
Limit Up to a specified number of traces.
Error Traces with at least one span with an error (error=true).

Understand Trace Query Results

A trace query:

  1. Finds the spans that match the description you specify.
  2. Finds the traces that contain at least one qualifying span.
  3. Uses duration thresholds (if you specified any) to filter the set of returned traces.

For example, you can query for traces that each have at least one member span that meets all of the following criteria:

  • Represents an operation called dispatch that is performed by a service called delivery.
  • Represents work done on a cluster called us-west-2.
  • Is associated with a custom tag env=production.

If you also specified a minimum (or maximum) duration, the query filters out any traces that are shorter (longer) than the threshold you specified.

Limited Result Set

Using the Limit filter to limit the number of returned traces can make your query complete faster. The trace query starts by returning the most recent traces. After reaching the limit, the query stops looking for more traces.

Note: The current time window for the Traces browser also implicitly limits by the result set. Traces are returned only if they contain a matching span and start within the current time window.

Sort the Result Set

You can sort a set of returned traces by selecting a sort order from the Sort By menu. For example:

  • Choose Most Recent to start with the traces that have the most recent start times.
  • Choose Longest First to start with the longest traces.
  • Choose Outliers to start with the traces whose duration is unusually long or unusually short.
  • Choose Most Spans to start with the traces that contain the largest number of spans.

If you both limit and sort the query results, sorting applies after limiting. For example, suppose you limit the number of returned traces to 50, and then sort the result set from shortest to longest. The sorted list includes only the 50 traces that were originally returned by the query. We do not first sort all traces containing a matching span, and then display the 50 shortest traces.

Note: If you’ve enabled a sampling strategy, results are found among the spans that have actually been ingested. The query does not search through spans before they’ve been sampled.

Use Query Editor (Power Users)

  1. Select Applications > Traces in the task bar to display the Traces browser.
  2. Click the icon to toggle to Query Editor:
    tracing query toggle
  3. Type a query that includes the traces() function:
    tracing query editor
  4. Click Search in the query bar to update the list of traces.