After your application sends trace data to Wavefront, you can examine that data in the Traces browser. By fine-tuning the trace query in the Traces browser, you find the traces that you’re interested in by describing the spans they must contain.
Get Started with Trace Queries
To query traces, select Applications > Traces and navigate to the Traces browser.
Query traces using a trace ID:
- Click Trace ID and enter the ID of the trace or traces you want to query.
- Click Search in the query bar.
- If you search for a trace after 7 days because Wavefront retains trace data for 7 days.
- If you search for a trace after 1 hour because you have enabled intelligent sampling for traces.
- If you use invalid query syntax.
Query traces using an operation:
- Click Operation and select an operation.
- Optionally, add one or more filters to refine your query.
- Click Search in the query bar.
Select an Operation
You select an operation to display traces with at least one span that represents the work done by that operation.
1. Select Applications > Traces to open the Traces browser.
2. Click the Operation selector to open the cascading menu.
3. Use the drill-down to select the application, service, and operation.
4. Either add filters (see below) or click Search to display results.
You can select
* instead of name components to select all operations in a service, or all operations in all services:
After you have selected an operation, you can optionally add filters to further describe the traces you want to see. You can add different types of filters in any order.
- Select an operation.
- Click Add Filter and select a filter type.
Choose This Filter Type To Return Tag Traces that have at least one span with a selected tag or a selected source (host). Use this type for indexed tags that your application uses, typically
source, and so on.
RawTag Traces that have at least one span with a specified tag, for example,
env="prod". Use this type for custom or unindexed span tags.
SpanLog Traces that have span logs.Note: Span logs are disabled by default and require Wavefront proxy version 5.0 or later. Contact email@example.com to enable the feature. Duration (Trace) Traces that have a specified minimum and/or maximum length (in milliseconds). Duration (Span) Traces that have a specified minimum and/or maximum span length (in milliseconds). Limit Up to a specified number of traces. Error Traces with at least one span that contains an error (
TraceId The trace or traces that match the trace ID or trace IDs.Note: If you select TraceId you can only search by the trace ID. You cannot filter an operation by a trace ID.
- Click next to the filter type and specify the filter setting. See the example given below.
- Click Add Filter to add another filter or click Search to display results.
You can remove an individual filter:
If you want to start the query all over, click Clear to remove the operation name and all filters:
Suppose you want to find traces that contain spans for an operation called
notify, which is called from the
notification service of the
beachshirts application. You’re interested only in seeing traces that are emitted from a particular source and that are longer than 30 milliseconds.
- Select the operation from the cascading Operation menu.
- Click Add Filter and select Tag.
- Click the Tags selector to open a cascading menu of tag types.
- Select the source tag type to display a menu of source names, and select the source name you want.
- Click Add Filter and select Duration.
- Click the min/max Duration selector to display fields for specifying durations.
- Type 30 in the Min Duration field and click Apply.
- Click Search to display the results.
View Trace Queries in Query Editor
- Construct a query with Query Builder as shown above.
- Toggle to the Query Editor to see what the corresponding functions look like.
Note: If you change a query using Query Editor, you cannot go back to Query Builder.
Trace Query Results
A trace query:
- Finds the spans that match the description you specify.
- Finds the traces that contain at least one matching span.
- Uses duration thresholds (if you specified any) to filter the set of returned traces.
For example, you can query for traces with at least one member span that meets the following criteria:
- Represents an operation called
dispatchthat is performed by a service called
- Represents work done on a cluster called
- Is associated with a custom tag
If you also specified a minimum (or maximum) duration, the query filters out any traces that are shorter (longer) than the threshold you specified.
Graphic Representation of a Returned Trace
Wavefront displays a bar for each trace that is returned by a trace query. The bar’s length visually indicates the trace’s duration. A blue area in the bar indicates where a matching span occurs in the trace, and how much of the trace it occupies:
How Wavefront Labels a Returned Trace
Each bar that is returned by a query represents a unique trace that has a unique trace ID. For readability, we label each trace by its root span, which is the first span in the trace. The trace’s label is the name of the operation that the root span represents.
For example, the two returned traces shown above both have a root span that represents work done by an operation called
ShoppingWebResource.getShoppingMenu. However, these root spans represent different executions of the operation, with different start times. Although the two root spans have the same operation name, they mark the beginning of two different traces.
Note: A trace’s root span might differ from the span that was specified in the query. For example, suppose you query for spans that represent
getAvailableColors operations. The query could return traces that begin with
ShoppingWebResource.getShoppingMenu, if those traces contain a
Limit and Sort the Result Set
The browser allows you to fine-tune what you see.
Use the Limit filter to limit the number of returned traces and make your query complete faster. The trace query starts by returning the most recent traces. After reaching the limit, the query stops looking for more traces.
Note: The current time window for the Traces browser also implicitly limits by the result set. Traces are returned only if they contain a matching span and start in the current time window.
Sort a set of returned traces by selecting a sort order from the Sort By menu. For example, choose Outliers to start with the traces whose duration is unusually long or unusually short. Or, choose Most Spans to start with the traces that contain the largest number of spans.
If you both limit and sort the query results, sorting applies after limiting. For example, suppose you limit the number of returned traces to 50, and then sort the result set from shortest to longest. The sorted list includes only the 50 traces that were originally returned by the query. We do not first sort all traces containing a matching span, and then display the 50 shortest traces.
Note: If you’ve enabled a sampling strategy, results are found among the spans that have actually been ingested. The query does not search through spans before they’ve been sampled.
Use Query Editor (Power Users)
Query Builder works well for many use cases, but sometimes Query Editor is your best option.