Learn how to query for Wavefront trace data.

After your application sends trace data to Wavefront, you can examine that data in the Traces browser. By fine-tuning the trace query in the Traces browser, you find the traces that you’re interested in by describing the spans they must contain.

Submitting Trace Queries

You query for traces by describing the spans they must contain. You can optionally specify duration thresholds that the returned traces must satisfy.

You submit queries and view the results in the Traces browser.

Use Query Builder

  1. Select Applications > Traces in the task bar to display the Traces browser.
    Query Builder is displayed by default.
  2. Use the trace query menus and fields to specify the characteristics to be matched. Each selection you make updates the list of traces.

    tracing query builder

Use Query Editor

  1. Select Applications > Traces in the task bar to display the Traces browser.
  2. Click the icon to toggle to Query Editor:
    tracing query toggle
  3. Type a query that includes the traces() function: tracing query editor

Trace Query Menus and Fields

Query Builder lets you use menus and fields to specify the traces you want to display.

Note: Certain menus correspond to tags that a developer specified while instrumenting the application code. An empty menu means that the code was instrumented without the corresponding tags.

  1. Display the Traces browser and make sure Query Builder is displayed. (It is displayed by default.)
  2. Select values from these menus to describe the logical characteristics that spans must match:
    MenuDescription
    Operation Match spans that represent work done by the selected operation. If you instrumented the code with application tags, this menu cascades so you can match spans that represent the work done by:
    • All operations in a selected service and application.
    • A selected operation in a selected service and application.
    Note: At a minimum you must select an application and a service.
    Filters Match spans from a selected source (host). If you instrumented the code with custom tags, this menu cascades so you can find spans associated with custom tags. Wavefront populates this menu based on the selection you made from the Operation menu.
  3. Select values from these menus to specify the physical characteristics that spans must match:
    MenuDescription
    Cluster Match spans from the selected cluster. A cluster is a named group of host machines. Wavefront populates this menu based on the selection you made from the Operation menu.
    Shard Match spans from a selected shard. A shard is a named subgroup of the hosts in a particular cluster, for example, a mirror. Wavefront populates this menu based on the selection you made from the Operation menu.
  4. Fill in one or both of these fields to return only traces of a minimum or maximum length:
    FieldDescription
    Min Duration Minimum number of milliseconds in a returned trace.
    Max Duration Maximum number of milliseconds in a returned trace.
  5. Choose a Limit to specify the maximum number of traces to display.

Example

Suppose you want to find traces that contain spans for an operation called dispatch, which is called from the delivery service of the beachshirts application. You’re interested only in traces that are longer than 30 milliseconds.

  1. Select the operation from the cascading Operation menu: tracing query builder menu
  2. Type 30 in the Min Duration field. tracing query builder menu2

Viewing the Corresponding Trace Query

Query Builder generates a query that includes the traces() function and one or more filtering functions. For example, you can:

  1. Construct a query with Query Builder as shown above.
  2. Toggle to the Query Editor to see what the corresponding functions look like. tracing query editor from builder

At this point, you can either continue to edit the query directly, or toggle back to Query Builder. Note: If you change a query using Query Editor, you cannot go back to Query Builder.

Understanding Trace Query Results

A trace query:

  1. Finds the spans that match the description you specify.
  2. Finds the traces that contain at least one qualifying span.
  3. Uses duration thresholds (if you specified any) to filter the set of returned traces.

For example, you can query for traces that each have at least one member span that meets all of the following criteria:

  • Represents an operation called dispatch that is performed by a service called delivery.
  • Represents work done on a cluster called us-west-2.
  • Is associated with a custom tag env=production.

If you also specified a minimum (or maximum) duration, the query filters out any traces that are shorter (longer) than the threshold you specified.

Graphic Representation of a Returned Trace

Wavefront displays a bar for each trace that is returned by a trace query. The bar’s length corresponds to the trace’s duration. A blue area in the bar indicates where a matching span occurs in the trace, and how much of the trace it occupies:

tracing query results

How Wavefront Labels a Returned Trace

Each bar that is returned by a query represents a unique trace that has a unique trace ID. For readability, we label each trace by its root span, which is the first span in the trace. The trace’s label is the name of the operation that the root span represents.

For example, the two returned traces shown above both have the label shopping: orderShirts. This is because both traces have a root span that represents the work done by the orderShirts operation in the shopping service. However, these root spans represent different executions of the orderShirts operation, with different start times. Although these two root spans have the same operation name, they mark the beginning of two different traces.

Note: A label such as shopping: orderShirts refers to the root span of a trace, which may be different from the span that was specified in the query. For example, suppose you query for spans that represent dispatch operations. The query could return traces that begin with orderShirts, if those traces contain a dispatch span.

Limiting the Result Set

You can limit the number of returned traces to make your query complete faster. The trace query starts by returning the most recent traces. After reaching the limit, the query stops looking for more traces.

Note: The current time window for the Traces browser also implicitly limits by the result set. Traces are returned only if they contain a matching span and start within the current time window.

Sorting the Result Set

You can sort a set of returned traces by selecting a sort order from the Sort By menu. For example:

  • Choose Most Recent to start with the traces that have the most recent start times.
  • Choose Longest First to start with the longest traces.
  • Choose Most Spans to start with the traces that contain the largest number of spans.

If you both limit and sort the query results, sorting applies after limiting. For example, suppose you limit the number of returned traces to 50, and then sort the result set from shortest to longest. The sorted list includes only the 50 traces that were originally returned by the query. We do not first sort all traces containing a matching span, and then display the 50 shortest traces.

Note: If you’ve enabled a sampling strategy, results are found among the spans that have actually been ingested. The query does not search through spans before they’ve been sampled.