Customize span level tags and alerts for RED metrics

Wavefront derives RED metrics for spans that have the application, service, cluster, shard, component, or operationName span tags by default. See Indexed and Unindexed Span Tags for details. If you want to filter RED metrics data using a span tag that is not a default span tag, you need to propagate it as a custom span tag to the RED metrics.

The following custom span tags are supported by default.

Custom span tagDescription
span.kind Filter spans based on the span type. The default value is none.
Example: client for a client-side span and server for a server-side span.
http.status_code Filter spans based on the error code.
Example: 404 or 500

Follow the steps given below to propagate custom span tags when sending data from your application. Once the data is in Wavefront, you can use queries to create custom dashboards that help you filter and view the information you need. Let’s look at a sample scenario that adds a custom span tag where you can compare the data in the production and staging environments.

  1. Create a custom span-level tag. Assume you have a span that has the env= span tag.

    The Tracing SDK provides a WavefrontTracer to create spans and send them to Wavefront. It also automatically generates and reports RED metrics from your spans. Add the following configuration when building the WavefrontTracer.

    Example:

    wfTracerBuilder.redMetricsCustomTagKeys(new HashSet<String>(Arrays.asList("env")));
             

    See the specific GitHub repository for language-specific examples on how to configure your application with the Wavefront OpenTracing SDK.

    If you are using Jaeger, you send data to Wavefront using Wavefront proxy. Add the configuration shown below to the <wavefront_config_path>/wavefront.conf file. See Paths to find out where the file is saved.

    traceDerivedCustomTagKeys=env
             

    If you are using Zipkin, you send data to Wavefront using Wavefront proxy. Add the configuration shown below to the <wavefront_config_path>/wavefront.conf file. See Paths to find out where the file is saved.

    traceDerivedCustomTagKeys=env
             

    Add the configuration shown below to your application's application.properties file.

    wavefront.tracing.red-metrics-custom-tag-keys=env
             

    Add the configuration shown below to the <wavefront_config_path>/wavefront.conf file. See Paths to find out where the file is saved.

    traceDerivedCustomTagKeys=env
             
  2. Save the changes, restart the application, and send data to Wavefront.
  3. Once the data is in Wavefront, create a chart that compares the data sent by each environment. Example: create a chart with custom span tags For details on querying for RED metrics, see the section above on RED Metrics Queries.