Learn about ways to send trace data to Wavefront from a 3rd party distributed tracing system.

You can collect traces with a 3rd party distributed tracing system, and send the trace data to Wavefront. Wavefront provides managed, highly scalable storage for your trace data, as well as RED metrics that are derived from the spans.

Suppose you have already instrumented your application using a 3rd party distributed tracing system such as Jaeger or Zipkin. You can continue using that system for application development, and then switch to a Wavefront proxy in production by changing a few configuration settings.

Note: If you have not yet instrumented your application for tracing, consider doing so with one or more Wavefront observability SDKs.

Tracing-System Integrations and Exporters

Wavefront supports integrations and also lets you specify custom tags.

Wavefront provides integrations with popular 3rd party distributed tracing systems. To get data flowing:

Using an integration is the simplest way - but not the only way - to send trace data to Wavefront from a 3rd part tracing system.

The Wavefront integration

  1. Configures your distributed tracing system to send trace data to a Wavefront proxy.
  2. The proxy processes the data and sends it to your Wavefront service.

Part of setting up the integration is to configure the Wavefront proxy to listen for the trace data on an integration-specific port.

Trace Data from an Integration

When you use a tracing-system integration, the Wavefront proxy receives the trace data that your application emits. If you configured your distributed tracing system to perform sampling, the proxy receives only the spans that the 3rd party sampler has allowed.

The Wavefront proxy:

  • Checks for required span tags on each received span, and adds them if needed.
  • Automatically derives RED metrics from the received spans.

Required Span Tags

Wavefront requires various span tags on well-formed spans. The following spans tags enable you to filter and visualize trace data from the different services in your instrumented application:

Tag NameDescriptionValue
application Name that identifies the application that emitted the span. By default, the name of your distributed tracing system, for example,Jaeger or Zipkin.
service Name that identifies the microservice that emitted the span. The service name that you specified to your distributed tracing system.

The proxy preserves any tags that you assigned through your distributed tracing system. You can explicitly instrument your code to add an application tag with a preferred application name.

Wavefront does not allow the mandatory span tags to have multiple values. Make sure that your application does not send spans with multiple application/service tags. For example, a span with two span tags service=notify and service=backend is invalid.

Note: Wavefront will ignore span tags with empty values.

Derived RED Metrics

Wavefront automatically derives RED metrics from the spans that are sent from the instrumented application services. RED metrics are measures of the request Rate, Errors, and Duration that are obtained from the reported spans. These metrics are key indicators of the health of your services, and you can use them as context to help you discover problem traces.

Wavefront stores the RED metrics along with the spans they are based on. For more details, see RED Metrics Derived From Spans.

Note: The level of detail for the RED metrics is affected by any sampling that is done by your 3rd party distributed tracing system. See Trace Sampling and RED Metrics from an Integration, below.

Custom Tags for RED Metrics

Starting with Wavefront proxy version 4.38, you can include custom tags for RED metrics. To do so, add:

traceDerivedCustomTagKeys=<comma-separated-custom-tag-keys>

to the proxy configuration at /etc/wavefront/wavefront-proxy/wavefront.conf. See Proxy Configuration Paths for details on the config file location.

Wavefront generates custom tags for the specified keys at the proxy.

For example, adding the following property causes the Wavefront proxy to generate 3 custom tags:

traceDerivedCustomTagKeys=tenant, env, location

Note: For faster performance, index only low-cardinality custom span tags. (A tag with low cardinality has comparatively few unique values that can be assigned to it.) See Indexed and Unindexed Span Tags for details.

Custom Application Names

Starting with Wavefront proxy version 4.38, you can set up custom application names for Jaeger and Zipkin. The process differs depending on your source, as follows.

Custom Application Names for Jaeger

You can specify custom application names at the level you need, like this:

  • Span-level tag: Add the application tag to all spans.
  • Process-level tag: Add the application tag as a Jaeger tracer tag, that is, a process tag.
  • Proxy-level tag: Add traceJaegerApplicationName=<application-name> in the proxy configuration at /etc/wavefront/wavefront-proxy/wavefront.conf. See Proxy Configuration Paths for details on the config file location.

The order of precedence is span level > process level > proxy level.

Custom Application Names for Zipkin

You can specify custom application names at the level you need, like this:

  • Span-level tag: Add tag application to all spans.
  • Proxy-level tag: Add traceZipkinApplicationName=<application-name> in the proxy configuration at /etc/wavefront/wavefront-proxy/wavefront.conf. See Proxy Configuration Paths for details on the config file location.

The order of precedence is span level > proxy level.

Visualizing Trace Data from an Integration

You view trace data from an integration using Wavefront charts and queries.

If you want context for identifying problem traces, you can start by viewing the derived RED metrics:

  1. Select Applications > Inventory in the task bar, and find the application (by default, Jaeger or Zipkin).
  2. Click on the application name, and find the service whose RED metrics you want to see.
  3. Click on the Details link for the service.
  4. Select an operation from one of the charts to examine the traces for that operation.

If you want to view trace data directly, you can start by submitting a trace query:

  1. Select Applications > Traces in the task bar.
  2. In the Traces browser, submit a trace query by selecting the operations and filters that describe the spans of interest.
  3. Click Search.

Trace Sampling and RED Metrics from an Integration

When you use a 3rd party distributed tracing system, you normally configure it to perform sampling. Doing so means that sampling occurs first, before the Wavefront proxy derives the RED metrics. The Wavefront proxy receives only a subset of the generated spans, and the RED metrics will reflect just that subset.

For more accurate RED metrics, you can disable the 3rd party sampling, and choose one of the following options instead:

The Wavefront proxy or Wavefront Tracer will auto-derive the RED metrics first, and then perform the sampling.

Alternatives to Integrations

Swap In a Wavefront Tracer

If you are using Jaeger (or some other tracing system that is compliant with the OpenTracing specification), you can replace the Jaeger Tracer with a Wavefront Tracer.

Swapping Tracers enables Wavefront to derive the RED metrics from the entire set of generated spans. In contrast, using the Jaeger integration causes the RED metrics to reflect just the subset of spans that are admitted by the Jaeger sampling.

For setup details, see the Wavefront OpenTracing SDK for your programming language.

Send Raw Trace Data

If Wavefront does not support an integration for your distributed tracing system, or if you are using your own proprietary tracing system, you can use a sender SDK to send raw trace data to Wavefront. With a sender SDK, you can write code that obtains the component values from your spans, and assembles those values into the Wavefront span format. The sender SDK also lets you configure your application to send the trace data to a Wavefront proxy or directly to the Wavefront service.

For SDK setup details, see the Wavefront sender SDK for your programming language.

Note: This technique does not automatically derive RED metrics from the spans.

Use the Wavefront OpenCensus Go Exporter

If you have instrumented your Go application with OpenCensus, you can use the Wavefront OpenCensus Go Exporter to push metrics, histograms, and traces into Wavefront. This exporter is built on the Wavefront sender SDK for Go.