Learn how to write traces() queries.

Summary

traces("<fullOperationName>" [,|and|or [not] <filterName>="<filterValue>"] ...)

traces(<filterName>="<filterValue>" [,|and|or [not] <filter2Name>="<filter2Value>"] ...)

traces(<spansExpression>, [<spansExpression])

Returns the traces that contain one or more qualifying spans, where a qualifying span matches the specified operation and span filters. Available only in the Query Editor in the Traces browser. Can be combined with one or more filtering functions.

Parameters

ParameterDescription
fullOperationName Full name of the operation that a qualifying span must represent. For example:
"beachshirts.delivery.dispatch" matches spans that represent calls to an operation named dispatch in the delivery service of the beachshirts application.
The general format is <application>.<service>.<operationName>, where each component consists of one or more period-delimited nodes. Replace operationName or serviceName with an asterisk * to match spans for any operation in any service.
filterName A span filter that a qualifying span must match. Span filters let you limit which spans to return traces for. You can optionally specify multiple span filters combined with Boolean operators.
filterValue Value accepted by a specified filterName.
spansExpression Expression that describes the set of qualifying spans. You typically specify a spans() query that is wrapped in a spans filtering function. For example, the following expression describes spans that qualify by being longer than 11 seconds:
highpass(11s, spans("beachshirts.styling.makeShirts")). You can specify more than one spansExpression.

Description

The traces() function returns a set of traces. Each trace contains at least one qualifying member span. A span qualifies if it matches the description you specify, which might consist of an operation name, one or more span filters, or a combination of these.

You submit a traces() function using the Query Editor in the Traces browser. Using the traces() function is a power-user alternative to using Query Builder.

traces() returns a trace:

  • If it contains a member span that matches the entire specified description.
  • If the description is a Boolean expression that combines multiple span filters, then the same span must satisfy all the filters.

    For example, the result set for traces(service=shopping and source=web1) includes any trace that has at least one member span that is associated with both of the tags service=shopping and source=web1. The result set does not include, e.g., a trace that has one member span with service=shopping and a different member span with source=web1.

To keep query execution manageable, combine traces() with a filtering function such as limit() in the same query.

To qualify spans based on their length, specify a spans() query that is wrapped in a spans filtering function such as highpass().

Examples

Assume your team has instrumented an application called beachshirts for tracing. This application has a service called styling that executes an operation called makeShirts. The application is deployed on several hosts in each of several clusters.

Note: To keep query execution manageable, these examples use traces() with the limit() function.

Display the traces that include spans for calls to makeShirt:

limit(100, traces("beachshirts.styling.makeShirts"))

Display the traces that include spans for any operation in the styling service:

limit(100, traces("beachshirts.styling.*"))

Display the traces that include spans for any operation in the beachshirts application executing on either of two specified hosts:

limit(100, traces("beachshirts.*.*" and (source="prod-app1" or source="prod-app10")))

Display the traces that include spans for calls to makeShirt that are shorter than 3 milliseconds:

limit(100, traces(lowpass(3ms, spans("beachshirts.styling.makeShirts"))))

Span Filters

Span filters allow you to limit which spans to return traces for. Span filters are key/value pairs associated with spans. Developers define the available span filters (also called application tags) as part of instrumenting the application code for tracing. If you did not instrument the code yourself, you can use autocompletion in the Query Editor to discover the available span filters.

The general format for a span filter is <filterName>="filterValue".

FilterDescriptionExample
application Name that identifies an instrumented application, for example, beachshirts. Matches spans that represent operations that are performed by the specified application. traces(application="beachshirts")
service Name that identifies an instrumented microservice, for example, delivery. Matches spans that represent operations that are performed by the specified microservice. traces(service="delivery")
cluster Name of a group of related hosts that serves as a cluster or region in which an instrumented application runs, for example, us-west-2. Matches spans that represent operations that are executed on the specified cluster. traces(cluster="us-west-2")
shard Name of a mirror or other subgroup of hosts within a cluster, for example, secondary. Matches spans that represent operations that are executed on the specified shard. traces(shard="secondary")
source Host or container that an instrumented application is running on. Matches spans that represent operations that are executed on the specified source. traces(source="prod-app1")
<spanTag> Custom tag defined in your application code. Matches spans that are associated with the specified custom span tag. traces(environment="production")
traceId Unique identifier for a trace. Returns the specified trace. traces(traceId="5b309723-fb83-4c9c-afb6-f0dc4b2bff13")

Filtering Functions

You can use filtering functions to provide additional levels of filtering for the results of the traces() function.

You can combine filtering functions. For example, to return up to 100 traces that are longer than 30 milliseconds, you can combine the limit(), highpass(), and traces() functions as follows:

  • limit(100, highpass(30ms, traces("beachshirts.delivery.dispatch")))

Note: To keep query execution manageable, combine traces() with at least the limit() function.

Spans Filtering FunctionDescription and Example
limit(<numberOfTraces>, <tracesExpression>) Limits the results of tracesExpression to include the specified numberOfTraces.
highpass(<traceDuration>, <tracesExpression>) Filters the results of tracesExpression to include only traces that are longer than traceDuration.
lowpass(<traceDuration>, <tracesExpression>) Filters the results of tracesExpression to include only traces that are shorter than traceDuration.