A trace shows you how a particular request propagates from one microservice to the next in a distributed application. The basic building blocks of a trace are its spans, where each span corresponds to a distinct invocation of an operation that executes as part of the request.
Spans are the fundamental units of trace data in Wavefront. This page provides details about the Wavefront format of a span, as well as the RED metrics that Wavefront automatically derives from spans. These details are mainly useful for developers who need to perform advanced customization.
Wavefront Span Format
A well-formed Wavefront span consists of fields and span tags that capture span attributes. These attributes enable Wavefront to identify and describe the span, organize it into a trace, and display the trace according to the service and application that emitted it. Some attributes are required by the OpenTracing specification and others are required by Wavefront.
Most use cases do not require you to know exactly how Wavefront expects a span to be formatted:
- When you instrument your application with a Wavefront OpenTracing SDK or a framework-instrumentation SDK, your application emits spans that are automatically constructed by the Wavefront Tracer. (You supply some of the attributes when you instantiate the ApplicationTags object required by the SDK.)
- When you instrument your application with a Wavefront sender SDK, your application emits spans that are automatically constructed from raw data you pass as parameters.
- When you instrument your application with a 3rd party distributed tracing system, your application emits spans that are automatically transformed by the integration you set up.
It is, however, possible to manually construct a well-formed span and send it either directly to the Wavefront service or to a TCP port that the Wavefront proxy is listening on for trace data. You might want to do this if you instrumented your application with a proprietary distributed tracing system.
<operationName> source=<source> <spanTags> <start_milliseconds> <duration_milliseconds>
Fields must be space separated and each line must be terminated with the newline character (\n or ASCII hex 0A).
"getAllUsers source=localhost traceId=7b3bf470-9456-11e8-9eb6-529269fb1459 spanId=0313bafe-9457-11e8-9eb6-529269fb1459 parent=2f64e538-9457-11e8-9eb6-529269fb1459 application=Wavefront service=auth cluster=us-west-2 shard=secondary http.method=GET 1552949776000 343"
||Yes||The string name that indicates the operation represented by the span.||Valid characters: a-z, A-Z, 0-9, hyphen ("-"), underscore ("_"), dot (".").
Length: less than 1024 characters.
||Yes||The string name of a host or container on which the represented operation executed.||Valid characters: a-z, A-Z, 0-9, hyphen ("-"), underscore ("_"), dot (".").
Length: less than 1024 characters.
||Yes||See Span Tags, below.|
||Yes||Start time of the span, expressed as epoch time elapsed since 00:00:00 Coordinated Universal Time (UTC) on January 1, 1970.||Whole number of epoch milliseconds or other units (see below).|
||Yes||Duration of the span.||Whole number of milliseconds or other units (see below). Must be greater than or equal to 0.|
Span tags are special tags associated with a span. Many of these span tags are required for a span to be valid. An application can be instrumented to include custom span tags as well. Custom tag names must not use the reserved span tag names listed in the following tables.
Note: The maximum allowed length for a combination of a span tag key and value is 254 characters (255 including the “=” separating key and value). If the value is longer, the span is rejected.
The following table lists span tags that contain information about the span’s identity and interrelationships.
|Span Tags |
||Yes||Unique identifier of the trace the span belongs to. All spans that belong to the same trace share a common trace ID.||UUID|
||Yes||Unique identifier of the span.||UUID|
||No||Identifier of the span’s dependent parent, if it has one. This tag is populated as the result of an OpenTracing
||No||Identifier of the span’s non-dependent parent, if it has one. This tag is populated as the result of an OpenTracing
The following table lists span tags that describe the architecture of the instrumented application that emitted the span. Wavefront uses these tags to aggregate and filter trace data at different levels of granularity. These tags correspond to the application tags you set through a Wavefront observability SDK.
|Span Tags |
||Yes||Name of the instrumented application that emitted the span.||String|
||Yes||Name of the instrumented microservice that emitted the span.||String|
||Yes||Name of a group of related hosts that serves as a cluster or region in which the instrumented application runs.
Specify cluster=none to indicate a span that does not use this tag.
||Yes||Name of a subgroup of hosts within the cluster.
Specify shard=none to indicate a span that does not use this tag.
Note: Additional span tags may be present, depending on how you instrumented your application. For example, the framework-instrumentation SDKs automatically use span tags like
http.method, and so on. You can find out about these tags in the README file for the the SDK on GitHub.
Time-Value Precision in Spans
A span has two time-value fields for specifying the start time (
start_milliseconds) and duration (
duration_milliseconds). We recommend that you express these values as in milliseconds, because those are the units that Wavefront uses for span storage and visualization. For convenience, you can specify time values in other units. Wavefront converts the values to milliseconds.
Wavefront requires that you use the same precision for both time values. Wavefront identifies the precision of the
start_milliseconds value, and interprets the
duration_milliseconds value using the same unit. The following table shows how to indicate the start-time precision:
|Precision for |
Start Time Values
|Number Format||Sample |
|Stored As |
|Seconds||Fewer than 13 digits||
||Multiplied by 1000|
(Thousandths of a second)
|13 to 15 digits||
(Millionths of a second)
|16 to 18 digits||
(Billionths of a second)
|19 or more digits||
Note: When specifying a span in Wavefront span format, make sure you adjust values as necessary so that the units match. For example, suppose you know a span started at
1533529977627 epoch milliseconds, and lasted for
3 seconds. In Wavefront span format, you could specify either of the following pairs of time values:
||(both values in seconds)|
||(both values in milliseconds)|
RED Metrics Derived From Spans
If you instrument your application with a tracing-system integration or with a Wavefront OpenTracing SDK, Wavefront derives RED metrics from the spans that are sent from the instrumented application. These out-of-the-box metrics are derived from your spans automatically, with no additional configuration or instrumentation on your part. You can use these metrics as context to help you discover problem traces.
RED metrics are measures of:
- Requests – the number of requests (spans) being served per second
- Errors – the number of failed requests (spans) per second
- Duration – per-minute histogram distributions of the amount of time that each request (span) takes
Note: RED metrics are also collected and sent by the framework-instrumentation SDKs. These SDKs report the RED metrics directly from the instrumented framework APIs, instead of deriving them from the reported spans. (Other metrics and histograms might be sent as well.)
Wavefront automatically generates charts to display the auto-derived RED metrics and histograms. To view these charts:
- Select Applications > Inventory in the Wavefront task bar. If necessary, scroll to find your application and its services.
- Click on the service you want to see metrics for.
- If you instrumented your application with a Wavefront SDK, look for the charts in the Overall section. (If you used a tracing-system integration, the charts are in the only section on the page.)
The auto-generated charts let you view the Request Rate, Error Rate, and Duration (P95) for the service, as well as the “top” operations each category: the most frequently invoked operations, the operations with the most errors, and the slowest operations. You can click on an operation in one of these charts to view the just the traces that contain that operation.
RED Metric Names
Wavefront constructs the names of the auto-derived RED metrics as shown in the following table. The name components
<operationName>) are string values that Wavefront obtains from the corresponding spans. If necessary, Wavefront modifies these strings to comply with Wavefront’s metric name format.
|Metric Name||Metric Type||Description|
||Counter||The number of times that the operation is invoked. Used in the Request Rate chart.|
||Counter||The number of invocations that are errors (i.e., spans with
||Wavefront histogram||The duration of each operation invocation, in microseconds, aggregated in one-minute intervals. Used in the Duration chart.|
Wavefront associates each auto-derived RED metric with point tags
operationName. Wavefront assigns the corresponding span values to these point tags. The span values are assigned without being modified.
Knowing the names of the auto-derived RED metrics lets you query and visualize these metrics just as you would any other metrics in Wavefront. For example, you can use the Duration metric in a histogram query to obtain percentiles other than the one displayed in the auto-generated chart.
Note: We recommend that you query for the auto-derived RED metrics using the point tags instead of metric names. The point tags preserve exact span values (and metric names might not).
Trace Sampling and Auto-Derived RED Metrics
If you have instrumented your application with a Wavefront observability SDK, Wavefront always derives the RED metrics before any sampling is performed. This is true when the sampling is performed by the SDK or when the sampling is performed by a Wavefront proxy. Consequently, Wavefront derives the RED metrics from a complete set of generated spans, so the metrics provide a highly accurate picture of your application’s behavior. However, if you click through a chart to inspect a particular trace, you might discover that the trace has not actually been ingested in Wavefront. You can consider configuring a less restrictive sampling strategy.
If you have instrumented your application using a 3rd party distributed tracing system, Wavefront derives the RED metrics after sampling has occurred. The Wavefront proxy receives only a subset of the generated spans, and the auto-derived RED metrics will reflect just that subset. See Trace Sampling and RED Metrics from an Integration.