Learn how to write traces() queries.

Summary

traces("<fullOperationName>" [and|or [not] <filterName>="<filterValue>"] ...)

traces(<filterName>="<filterValue>" [and|or [not] <filterName>="<filterValue>"] ...)

Returns the traces that contain one or more qualifying spans, where a qualifying span matches the specified operation and filters. Available only in the Query Editor in the Traces browser. Can be combined with one or more filtering functions.

Parameters

ParameterDescription
fullOperationName Full name of the operation that a qualifying span must represent. For example, specify "beachshirts.delivery.dispatch" to match the spans that represent calls to an operation named dispatch in the delivery service of the beachshirts application.
The general format of a fullOperationName is <application>.<service>.<operationName>, where each component consists of one or more period-delimited components. Replace operationName or serviceName with an asterisk * to match spans for any operation in any service.
filterName A span filter that a qualifying span must match. Span filters let you limit which spans to return traces for. You can optionally specify multiple span filters combined with Boolean operators (and, and not, or).
filterValue Value accepted by a specified filterName.

Description

The traces() function returns a set of traces, where each trace contains at least one qualifying member span. A span qualifies if it matches the description you specify, which might consist of an operation name, one or more span filters, or a combination of these.

You submit a traces() function using the Query Editor in the Traces browser. Using the traces() function is a power-user alternative to using Query Builder.

traces() returns a trace if it contains a member span that matches the entire specified description. If the description is a Boolean expression that combines multiple span filters, then the same span must satisfy all of the filters in the expression. For example, the result set for traces(service=shopping and source=web1) includes any trace that has at least one member span that is associated with both of the tags service=shopping and source=web1. The result set does not include, e.g., a trace that has one member span with service=shopping and a different member span with source=web1.

To keep query execution manageable, combine traces() with a filtering function such as limit() in the same query.

You can specify the length of the qualifying spans by including a spans() expression as the fullOperationName parameter.

Examples

Assume your team has instrumented an application called beachshirts for tracing. This application has a service called styling that executes an operation called makeShirts. The application is deployed on several hosts in each of several clusters.

Note: To keep query execution manageable, these examples use traces() with the limit() function.

To display the traces that include spans for calls to makeShirt:

  • limit(100, traces("beachshirts.styling.makeShirts"))

To display the traces that include spans for any operation in the styling service:

  • limit(100, traces("beachshirts.styling.*"))

To display the traces that include spans for any operation in the beachshirts application executing on either of two specified hosts:

  • limit(100, traces("beachshirts.*.*" and source=prod-app1 or source=prod-app10))

Span Filters

Span filters allow you to limit which spans to return traces for. Span filters are key/value pairs associated with spans. Developers define the available span filters (also called application tags) as part of instrumenting the application code for tracing. Even if you did not instrument the code yourself, you can use autocompletion in the Query Editor to discover the available span filters.

The general format for a span filter is <filterName>="filterValue".

FilterDescriptionExample
application Name that identifies an instrumented application, for example, beachshirts. Matches spans that represent operations that are performed by the specified application. traces(application="beachshirts")
service Name that identifies an instrumented microservice, for example, delivery. Matches spans that represent operations that are performed by the specified microservice. traces(service="delivery")
cluster Name of a group of related hosts that serves as a cluster or region in which an instrumented application runs, for example, us-west-2. Matches spans that represent operations that are executed on the specified cluster. traces(cluster="us-west-2")
shard Name of a mirror or other subgroup of hosts within a cluster, for example, secondary. Matches spans that represent operations that are executed on the specified shard. traces(shard="secondary")
source Host or container that an instrumented application is running on. Matches spans that represent operations that are executed on the specified source. traces(source="prod-app1")
<spanTag> Custom tag defined in your application code. Matches spans that are associated with the specified custom span tag. traces(environment="production")
traceId Unique identifier for a trace. Returns the specified trace. traces(traceId="5b309723-fb83-4c9c-afb6-f0dc4b2bff13")

Filtering Functions

You can use filtering functions to provide additional levels of filtering for the results of the traces() function.

Note: To keep query execution manageable, combine traces() with at least the limit() function.

Each filtering function has a tracesExpression parameter, which can be a traces() function or one of the other filtering functions. For example, to return up to 100 traces that are longer than 30 milliseconds, you can combine the limit(), highpass(), and traces() functions as follows:

  • limit(100, highpass(30ms, traces("beachshirts.delivery.dispatch")))
Spans Filtering FunctionDescription and Example
limit(<numberOfTraces>, <tracesExpression>) Limits the results of tracesExpression to include the specified numberOfTraces.
Note: Because the ordering of traces is unpredictable, you cannot use limit() to page through a set of results to obtain the next group of traces.

Example: Limit the set of returned traces to 50:
limit(50, traces("beachshirts.styling.makeShirts"))
highpass(<traceDuration>, <tracesExpression>) Filters the results of tracesExpression to include only traces that are longer than traceDuration. Specify traceDuration as an integer number of milliseconds, seconds, minutes, hours, days or weeks (1ms, 1s, 1m, 1h, 1d, 1w).

Example: Limit the result set to include traces that are longer than 3 seconds:
highpass(3s, traces("beachshirts.styling.makeShirts"))
lowpass(<traceDuration>, <tracesExpression>) Filters the results of tracesExpression to include only traces that are shorter than traceDuration. Specify traceDuration as an integer number of milliseconds, seconds, minutes, hours, days or weeks (1ms, 1s, 1m, 1h, 1d, 1w).

Example: Limit the result set to include traces that are shorter than 10 milliseconds:
lowpass(10ms, traces("beachshirts.styling.makeShirts"))