Learn how to write spans() queries.

Summary

spans("<fullOperationName>" [,|and|or [not] <filterName>="<filterValue>"] ...)

spans(<filterName>="<filterValue>" [,|and|or [not] <filter2Name>="<filter2Value>"] ...)

Returns the spans that match the specified operation and filters. You use spans() as a parameter of the traces() function, typically after combining spans() with one or more spans filtering functions.

Parameters

ParameterDescription
fullOperationName Full name of the operation that each matching span must represent. For example, specify "beachshirts.delivery.dispatch" to match the spans that represent calls to an operation named dispatch in the delivery service of the beachshirts application.
The general format of a fullOperationName is <application>.<service>.<operationName>, where each component consists of one or more period-delimited components. Replace operationName or serviceName with an asterisk * to match spans for any operation in any service.
filterName A span filter that each matching span must match. Span filters let you limit which spans to return traces for. You can optionally specify multiple span filters combined with Boolean operators.
filterValue Value accepted by a specified filterName.

Description

The spans() function finds spans that match the description you specify. You describe the spans of interest by providing an operation name, one or more filters, or a combination of these, to specify the characteristics that the spans must match.

You use spans() by including it as a parameter of the traces() function. Doing so allows you to filter traces according to the duration of a particular span. For example, the following query returns a trace only if it has at least one span that both represents a makeShirts operation and lasts longer than 11 seconds:

limit(100, traces(highpass(11s, spans("beachshirts.styling.makeShirts"))))

This is different from a query that wraps highpass around traces(). For example, the following query returns a trace only if the entire end-to-end trace is longer than 11 seconds.

limit(100, highpass(11s, traces(spans("beachshirts.styling.makeShirts"))))

Note: the following two queries are equivalent:

limit(100, traces("beachshirts.styling.makeShirts"))          //  spans() is implicit
limit(100, traces(spans("beachshirts.styling.makeShirts")))   //  spans() is explicit

Examples

Assume your team has instrumented an application called beachshirts for tracing. This application has a service called styling that executes an operation called makeShirts. The application is deployed on several hosts in each of the several clusters.

Note: To keep query execution manageable, these examples use traces() with the limit() function.

  • To display the traces that include long spans for calls to makeShirt:
    limit(100, traces(highpass(11s, spans("beachshirts.styling.makeShirts"))))
    
  • To display the traces that include short spans for any operation in the styling service:
    limit(100, traces(lowpass(3ms, spans("beachshirts.styling.*"))))
    
  • To display the traces that include spans for any operation in the beachshirts application executing on either of two specified hosts:
    limit(100, traces(spans("beachshirts.*.*" and (source="prod-app1" or source="prod-app10"))))
    

Span Filters

Span filters allow you to limit which spans to return traces for. Span filters are key/value pairs associated with spans. Developers define the available span filters (also called application tags) as part of instrumenting the application code for tracing. Even if you did not instrument the code yourself, you can use autocompletion in the Query Editor to discover the available span filters.

The general format for a span filter is <filterName>="filterValue".

FilterDescriptionExample
application Name that identifies an instrumented application, for example, beachshirts. Matches the spans that represent operations that are performed by the specified application. spans(application="beachshirts")
service Name that identifies an instrumented microservice, for example, delivery. Matches the spans that represent operations that are performed by the specified microservice. spans(service="delivery")
cluster Name of a group of related hosts that serves as a cluster or region in which an instrumented application runs, for example, us-west-2. Matches the spans that represent operations that are executed on the specified cluster. spans(cluster="us-west-2")
shard Name of a subgroup of hosts within a cluster, for example, secondary. Matches the spans that represent operations that are executed on the specified shard. spans(shard="secondary")
source Host or container that an instrumented application is running on. Matches the spans that represent operations that are executed on the specified source. spans(source="prod-app1")
traceId Unique identifier for a trace. Matches all spans that belong to the specified trace. spans(traceId="5b309723-fb83-4c9c-afb6-f0dc4b2bff13")
<spanTag> Custom tag defined in your application code. Matches spans that are associated with the specified custom span tag. spans(environment="production")

Spans Filtering Functions

You can use spans filtering functions to provide additional levels of filtering for the set of spans that are matched by the spans() function.

Each spans filtering function has a spansExpression parameter, which can be a spans() function or one of the other spans filtering functions.

Spans Filtering FunctionDescription and Example
highpass(<spanDuration>, <spansExpression>) Limits the set of spans that are matched by spansExpression to include only spans that are longer than spanDuration.
lowpass(<spanDuration>, <spansExpression>) Limits the set of spans that are matched by spansExpression to include only spans that are shorter than spanDuration.

Spans Operators

Use the following operators to get details or find the relationship between services in an application and their operations using the spans() functions.

Operator Description
from Returns spans that are a child of or directly follow a given span
<child_spansExpression>.from(<parent_spansExpression>)

Example:
Search for spans in the beachshirts application and get the following spans:
  • Spans where the shopping service is the parent span of the inventory service (or spans where the inventory service is the child of the shopping-service )
  • Spans where the inventory service directly follow after the shopping-service.
spans(beachshirts.inventory.*).from(spans(beachshirts.shopping.*))
Search for spans where the spans from the shopping service are longer than 1000 milliseconds and are the parent spans of the inventory service.
spans(beachshirts.inventory.*).from(highpass(1000, spans(beachshirts.shopping.*)))
childOf Deprecated OpenTracing operator.
Returns spans that are a child of a given span. This concept is a result of the OpenTracing ChildOf relationship.
<child_spansExpression>.childOf(<parent_spansExpression>)

Example:
Search for spans in the beachshirts application and only get the spans where the shopping service is the parent span of the inventory service (or spans where the inventory service is the child of the shopping service).
spans(beachshirts.inventory.*).childOf(spans(beachshirts.shopping.*))
followsFrom Returns spans that directly follow a given span. This concept is a result of the OpenTracing followsFrom relationship.
<child_spansExpression>.followsFrom(<parent_spansExpression>)

Example: Search for spans in the beachshirts application and get the spans from the inventory service that directly follow the shopping service.
spans(beachshirts.inventory.*).followsFrom(spans(beachshirts.shopping.*))