You can collect traces with a 3rd party distributed tracing system, and send the trace data to Wavefront. Wavefront provides managed, highly scalable storage for your trace data, as well as RED metrics that are derived from the spans.
Suppose you have already instrumented your application using a 3rd party distributed tracing system such as Jaeger or Zipkin. You can continue using that system for application development, and then switch to a Wavefront proxy in production by changing a few configuration settings.
Wavefront provides integrations with several popular 3rd party distributed tracing systems:
The Wavefront integration configures your distributed tracing system to send trace data to a Wavefront proxy. The proxy, in turn, processes the data and sends it to your Wavefront service. Part of setting up the integration is to configure the Wavefront proxy to listen for the trace data on an integration-specific port.
Using an integration is the simplest way - but not the only way - to send trace data to Wavefront from a 3rd part tracing system.
Trace Data from an Integration
When you use a tracing-system integration, the Wavefront proxy receives the trace data that your application emits. If you configured your distributed tracing system to perform sampling, the proxy receives only the spans that the 3rd party sampler has allowed.
The Wavefront proxy:
- Checks for required span tags on each received span, and adds them if needed.
- Automatically derives RED metrics from the received spans.
Required Span Tags
Wavefront requires various span tags on well-formed spans. The following spans tags enable you to filter and visualize trace data from the different services in your instrumented application:
||Name that identifies the application that emitted the span.||By default, the name of your distributed tracing system, i.e.,
||Name that identifies the microservice that emitted the span.||The service name that you specified to your distributed tracing system.|
Note: The proxy preserves any tags that you assigned through your distributed tracing system. You can explicitly instrument your code to add an
application tag with a preferred application name.
Auto-Derived RED Metrics
When you use a tracing-system integration, Wavefront automatically derives RED metrics from the spans that are sent from the instrumented application services. RED metrics are measures of the Request rate, Error rate, and Duration of the operations represented by your spans. You can use these metrics as context to help you discover problem traces.
Wavefront stores the RED metrics along with the spans they are based on. For more details, see RED Metrics Derived From Spans.
Note: The level of detail for the RED metrics is affected by any sampling that is done by your 3rd party distributed tracing system. See Trace Sampling and RED Metrics from an Integration, below.
Visualizing Trace Data from an Integration
You view trace data from an integration using Wavefront charts and queries.
If you want context for identifying problem traces, you can start by viewing the auto-derived RED metrics:
- Select Applications > Inventory in the task bar, and find the application (by default,
- Click on the service whose auto-derived RED metrics you want to see.
- Select an operation from one of the charts to examine the traces for that operation.
If you want to view trace data directly, you can start by submitting a trace query:
- Select Applications > Traces in the task bar.
- In the Traces page, build a trace query by selecting the filters that describe the spans of interest. At a minimum, you must select the application (by default,
Zipkin) from the Operation menu.
Trace Sampling and RED Metrics from an Integration
When you use a 3rd party distributed tracing system, you normally configure it to perform sampling. Doing so means that sampling occurs first, before the Wavefront proxy derives the RED metrics. The Wavefront proxy receives only a subset of the generated spans, and the auto-derived RED metrics will reflect just that subset.
For more accurate RED metrics, you can disable the 3rd party sampling, and choose one of the following options instead:
- Set up sampling through the Wavefront proxy. You need proxy version 4.36 or later.
- Swap in a Wavefront Tracer and configure it to perform sampling.
The Wavefront proxy or Wavefront Tracer will auto-derive the RED metrics first, and then perform the sampling.
Swapping In a Wavefront Tracer
If you are using Jaeger (or some other tracing system that is compliant with the OpenTracing specification), you can replace the Jaeger Tracer with a Wavefront Tracer.
Swapping Tracers enables Wavefront to derive the RED metrics from the entire set of generated spans. In contrast, using the Jaeger integration causes the auto-derived RED metrics to reflect just the subset of spans that are admitted by the Jaeger sampling.
See the Wavefront OpenTracing SDK for your programming language:
Sending Raw Trace Data
If Wavefront does not support an integration for your distributed tracing system, or if you are using your own proprietary tracing system, you can use a sender SDK to send raw trace data to Wavefront. With a sender SDK, you can write code that obtains the component values from your spans, and assembles those values into the Wavefront span format. The sender SDK also lets you configure your application to send the trace data to a Wavefront proxy or directly to the Wavefront service.
For SDK setup details, see the Wavefront sender SDK for your your programming language:
Note: This technique does not automatically derive RED metrics from the spans.