Best practices for collecting and sending trace data from an application to Wavefront.

The best practices on this page help you get optimal results from instrumenting your application for tracing.

Planning for Tracing

  1. Learn about traces and spans. Traces represent end-to-end requests across services, and consist of spans, which represent calls to individual operations. See Tracing Basics for basic tracing concepts, and see https://opentracing.io for comprehensive discussion and details.

  2. Inventory your application to answer these questions:
    • Which services belong to your application? Which ones participate in the most critical requests?
    • What programming language or languages are these services written in?
    • Are any services built with open-source component frameworks? Which frameworks?
  3. Choose your Wavefront instrumentation support.
    • Instrument each service with the Wavefront OpenTracing SDK in the service’s language. Get a head start by using any Wavefront framework SDKs that are available for your service’s language and frameworks. Augment the framework SDKs with the Wavefront OpenTracing SDK.
    • If you have already instrumented your application with a 3rd party distributed tracing system such as Jaeger or Zipkin, set up a Wavefront integration.
    • Use consistent instrumentation, either Wavefront SDKs or a 3rd party tracing systemm for all services that participate in the same trace. Otherwise, spans cannot link to each other across service boundaries. You can intermix different Wavefront SDKs in different programming languages.

Best Practices for Sending Trace Data Through a Wavefront Proxy

Large-scale applications should use a Wavefront proxy to send trace data to Wavefront. A proxy is required with the Jaeger and Zipkin integrations.

Best Practices for Wavefront Observability SDKs

  1. Install and configure the Wavefront proxy with listener ports for metrics, histograms, and trace data. All three types of data are necessary for displaying RED metrics derived from spans.

    Note: Be sure to configure the proxy with the histogramDistListener= property. You might overlook this property if you are already using a proxy that is configured for metrics.

  2. Configure your application code to send data via the Wavefront proxy:

    • Set up a Wavefront sender that can find the Wavefront proxy host.
      • Configure the Wavefront sender with the same listener ports you set for the Wavefront proxy.
      • Instantiate a single Wavefront sender per process and share it among SDKs.
    • Java example: Instantiate a singleton WavefrontSender
  // Create the builder with the proxy hostname or address
  WavefrontProxyClient.Builder wfProxyClientBuilder = new WavefrontProxyClient.Builder(proxyHostName);

  // Set the listening ports for metrics, histograms, and trace data
  wfProxyClientBuilder.metricsPort(2878);        // same as pushListenerPorts= in proxy config
  wfProxyClientBuilder.distributionPort(2878);   // same as histogramDistListenerPorts= in proxy config
  wfProxyClientBuilder.tracingPort(30_000);      // same as traceListenerPorts= in proxy config

  // Create the WavefrontProxyClient
  WavefrontSender wavefrontSender = wfProxyClientBuilder.build();

Note: Complete setup steps are in the README file for your Wavefront SDK on GitHub.

Best Practices for 3rd Party Tracing Systems

Follow the integration’s setup steps to configure the Wavefront proxy with a special port for the integration. The integration implicitly instantiates and configures an internal Wavefront sender for you.

Application Inventory Best Practices

Think of your instrumented application as a hierarchic inventory of constructs. Wavefront requires that you assign a name to the constructs at each level. Consider this example:

ConstructExample NamesDescription
Application beachshirts Top-level construct that identifies a set of interacting services.
Service "delivery", "packaging", "printing" Constructs that implement the operations to be traced.
Component "grpc", "django", "jersey" Open-source component frameworks that services might be built with.

Wavefront uses these names as span tag values, as filters for traces, as components in RED metric names, as UI labels, and as qualifiers for operation names, for example, beachshirts.delivery.dispatch.

Guidelines for Choosing Application Construct Names

  • Choose a string name for each construct. Names at the same level must be unique.
    • Example: Specify only one application named beachshirts in a Wavefront instance, and only one service named delivery in a given application.
    • Note: Duplicate application, service, or component names might result in incorrect RED metrics.
  • Choose logical names that clearly map to your applications and services. Logical names might be simpler, more readable versions of code identifiers.

Best Practices for Wavefront Observability SDKs

  • Set up an Application tags object in each service to define logical names for the application constructs.
    • Specify the logical application and service names that apply to the service. Optionally include logical cluster and shard names, if you want to use the physical topology to filter your data.
    • Define a custom tag called component if you are using the Wavefront OpenTracing SDK.
      Note: Other Wavefront SDKs automatically define component for you.
  • Java example: Instantiate ApplicationTags for the delivery service
  String application = "beachshirts";
  String service = "delivery";
  String cluster = "us-west-2";
  String shard = "secondary";

  Map<String, String> customTags = new HashMap<String, String>() { { 
    put("location", "Oregon");
    put("env", "Staging");
    put("component", "Jersey");
   } } ;

  ApplicationTags applicationTags = new ApplicationTags.Builder(application, service).
      cluster(cluster).      
      shard(shard).      
      customTags(customTags). 
      build();

Note: Complete setup steps are in the README file for your Wavefront SDK on GitHub.

Best Practices for 3rd Party Tracing Systems

Wavefront automatically assigns standard application names, service names, and component names, based on the tags that are set for your 3rd party distributed tracing system.

Source Names Best Practices

A source is a host, container, Kubernetes pod, instance, or any other unique origination point for application code that is sending a span to Wavefront.

Wavefront requires that you choose unique names for the sources that send spans. Wavefront uses source names to filter traces and to define RED metrics.

Guidelines for Choosing Source Names

  • Choose a unique string name for every source that will send spans to Wavefront.
    • For example, use a machine’s IP address or a descriptive logical name.
    • If you use a machine’s host name (the default), make sure all machines have unique host names. Use logical names to distinguish machines with the same host names in different data centers.
    • Note: Duplicate source names might result in incorrect RED metrics.
  • See also the guidelines for choosing source names for metric data and histograms.

Best Practices for Wavefront Observability SDKs

  • Set up a WavefrontSpanReporter object to define a source name:
    • Specify the source name explicitly, or leave it unspecified to automatically use the host name. Make sure the host name is unique.
    • If your Wavefront SDK defines additional reporters, specify the same source name in each one. All reporter objects for a particular service must specify the same source.
  • Java example: Build a WavefrontSpanReporter that reports from a source called wavefront-tracing-example.
  // Create a WavefrontProxyClient
  WavefrontSender wavefrontSender = ... 

  Reporter wfSpanReporter = new WavefrontSpanReporter.Builder().
    withSource("wavefront-tracing-example"). 
    build(wavefrontSender);

Note: Complete setup steps are in the README file for your Wavefront SDK on GitHub.

Span Names Best Practices

Spans are the building blocks of traces. Each span corresponds to a particular invocation of an operation. For example, a span might represent a specific method call such as getShoppingMenu(menu_id=123).

The OpenTracing standard requires that you choose names for the spans that your instrumented application creates and sends. Wavefront uses span names as part of the data format of each span, as filters for traces, and as the basis for RED metrics, for example, to report the number of calls to the getShoppingMenu method per minute.

Guidelines for Choosing Span Names

  • When you name a span, use the name of the operation or method being invoked as the span name.
  • Adopt naming conventions so that you’ll end up with no more than 1000 unique span-source pairs.
  • Add extra detail as span tags, instead of incorporating that detail into the span name.
  • Example: Suppose you are instrumenting a call to getShoppingMenu(menu_id=123).
    • A good choice for the span name is getShoppingMenu. You might add a custom span tag menu_id=123 to preserve the menu ID detail.

    • A poor choice for the span name is getShoppingMenu_123. If you incorporate the menu ID detail directly into the span names, you might end up with unique span names like getShoppingMenu_122, getShoppingMenu_123, getShoppingMenu_124, … getShoppingMenu_nnn, which all represent calls to the same piece of code. The result is a cardinality explosion!

Best Practices for 3rd Party Tracing Systems

If you are using Jaeger, verify that the number of generated span names will result in fewer than 1000 unique span-source pairs. If necessary, fix Jaeger instrumentation to produce fewer span names.

Custom Span Tags Best Practices

You can define custom span tags to let you query and filter for particular subsets of trace data.

Wavefront uses indexing to optimize the speed of querying and filtering with tags. By default, Wavefront indexes all point tags and certain built-in span tags. Indexing for custom span tags is available on request.

Guidelines for Defining Custom Span Tags

  • Keep the cardinality of custom span tags low (< 1000 values per tag) and contact Wavefront support to request indexing for those span tags. Indexing is available only for low-cardinality custom span tags.

  • If you are using a Wavefront OpenTracing SDK, be sure to define a span tag called component. Wavefront uses the component, application, and service tags to populate the Application Services page and each service-specific page.

Instrumentation Best Practices

The goal of instrumentation is to instrument enough methods to produce traces that can help you troubleshoot errors or pinpoint bottlenecks. You usually do this in successive passes.

  1. Go wide: Produce end-to-end traces across all services.
    • Focus on the entry/exit points of your services. Instrument each inbound and outbound request to report spans.
  2. Go deep: Produce traces that contain a deep, meaningful hierarchy of spans.
    • Identify the classes and methods that implement significant operations within each service, and instrument those methods.

Best Practices for Wavefront Observability SDKs

  • Automate as much as possible by using Wavefront framework SDKs, if any exist for your languages and framework. Then use a Wavefront OpenTracing SDK to instrument operations that are not handled by a framework SDK.

  • Limit the number of spans in a trace to < 1000.

  • Java example: Instantiate a singleton WavefrontTracer, pass it to each class, and use it in each method of interest:

  Span span = tracer.buildSpan("<name>").start();
  try {
    //app logic
  } catch (HandledException e) {
    // handle exception logic
    Tags.ERROR.set(span, true);
  } finally {
    span.finish();
  }

Sampling Best Practices

A large-scale web application can produce a high volume of traces. Consider limiting the volume of trace data using a head-based sampling strategy.