A cloud-scale web application generates a very large number of traces. Wavefront supports sampling to reduce the volume of stored trace data.
How It Works
Let’s look at the following scenarios to understand how sampling works in Wavefront:
Not all the trace data that you send to Wavefront are useful. Once traces are sent to Wavefront, Wavefront identifies the important traces and those that add value to you and retains them. This process is known as Intelligent Sampling.
However, when intelligent sampling is on, you might not see some traces when you search for them on the traces browser. If you want to have specific trace data in Wavefront and don’t want the traces to be discarded, use Sampling Policies. With a sampling policy in place, Wavefront does not perform intelligent sampling on the data sampled by the sampling policy
To see the number of spans stored per second after a sampling policy is created, see Track Volume of Trace Data Stored in Wavefront
Benefits of Sampling Data
Sampling has the following advantages:
- Reduce the amount of storage required for trace data, and lower your monthly costs.
- Only see traces that add value to you.
- Limit the performance impact on network bandwidth and application response times.
Wavefront Intelligent Sampling
Wavefront automatically performs intelligent sampling to reduce the volume of ingested traces. The goals of intelligent sampling are to retain traces that are likely to be informative, and to discard traces that are redundant or otherwise not worth inspecting.
Intelligent sampling gives preference to:
- Traces that are abnormally long, as compared to other traces for the same endpoint.
- Traces that contain at least one individual span that is abnormally long, as compared to other spans for the same operation.
- Traces that contain at least one span in which an error occurred.
Wavefront uses proprietary algorithms to decide which traces to retain (sample) and which traces to discard (not sample). When analyzing whether a trace is worth retaining, Wavefront compares the trace’s characteristics to a historical context that is composed of similar traces. The historical context is based on the RED metrics that Wavefront derives from the entire set of trace data that your application has emitted before any sampling occurs. This enables Wavefront to determine whether an analyzed trace is a true outlier.
Intelligent sampling applies to entire traces after Wavefront receives them. If you have set up an explicit sampling strategy, then the output of your explicit sampling strategy is the input to intelligent sampling.
Intelligent sampling is performed by the Wavefront service itself, not by the proxy or by an instrumented application. Consequently, intelligent sampling does not place any additional processing burden on your proxies or applications. Intelligent sampling does not add to your total cost of operation (TCO). If you already use one or more proxies to ingest your time-series data, you can start ingesting and sampling trace data without adding more hardware to support more proxies.
debug=true. Make sure to remove the annotation once you are done troubleshooting and don’t overuse the annotation as Wavefront intelligent sampling gives preference to unique spans. For details on adding span tags via the Wavefront proxy, see Proxy Preprocessor Rules.
You can monitor your span storage by checking the following internal metrics. If you have set up sampling, these metrics report the number of spans after sampling takes place.
||Number of spans per second being sent via a Wavefront proxy.|
||Number of spans per second being sent directly to the Wavefront service (direct ingestion).|
If you can’t find traces in Wavefront because Intelligent Sampling discarded them, create a sampling policy to let Wavefront know that you want to keep specific spans in Wavefront. Sampling policies impact the volume of spans ingested by Wavefront and can affect your costs. See your Service Description for cost details.
See Managing Sampling Policies for details.
Track the Volume of Trace Data Stored in Wavefront
A sampling policy affects your costs because more data maybe sent to Wavefront. To see the number of spans you store in Wavefront with the sampling policies:
- Click Dashboards > All Dashboards.
- Search for the Wavefront Service and Proxy Data dashboard and click it to navigate to the dashboard.
- On the dashboard, search for the Spans Sampled by Policies Per Second chart under Proxies overview.
You see the number of spans stored per second.
Explicit Sampling Strategies
An explicit sampling strategy is a mechanism for selecting which traces to forward to Wavefront. Wavefront supports the following explicit sampling strategies:
Explicit Sampling Strategy Overview
|Rate-based sampling||Sends N percent of the generated traces to Wavefront. Sometimes called probabilistic sampling. For example, a sampling rate of 10% causes 1 out of 10 traces to be sent and ingested.|
|Duration-based sampling||Sends spans to Wavefront only if they are longer than N milliseconds. For example, a sampling duration of 45 sends spans to Wavefront only if they are longer than 45 milliseconds.|
Ways to Set Up Explicit Sampling Strategies
You can set up an explicit sampling strategy using either of the following methods:
Choose the Wavefront proxy for sampling when you want to:
- Use a single sampling strategy to coordinate the sampling for all applications that use the same proxy.
- Configure sampling with minimal effort.
- Improve the likelihood of ingesting complete traces.
Choose sampling in your instrumented code when you want to:
- Reduce the performance impact of span reporting on your application.
- Use direct ingestion (no proxy).
- Configure sampling on a per-process basis, for example, when you expect spans from the services in different processes to have different characteristics.
Complete vs. Partial Traces
An ingested trace can be complete (a trace ingested with all of its member spans) or partial (a trace that is missing one or more spans). The completeness of the traces in a sample depends in part on the sampling strategy:
Rate-based sampling attempts to send complete traces. That is, the sampler selects the specified percentage of trace IDs, and then sends all of the spans that belong to each selected trace.
Duration-based sampling considers only individual spans. That is, the sampler selects all spans of an appropriate duration, regardless of whether they form complete traces.
Partial traces can also occur in the following situations:
- If a span contains an error. Each such span is sent individually, without the other spans in the same trace.
- If a trace has spans from multiple services, and you set up different sampling rates for those services.
Result of Combining Explicit Sampling Strategies
You can combine rate-based sampling and duration-based sampling in the same service. Doing so causes Wavefront to ingest the union of the spans that are selected by each sampler.
For example, suppose you set the sampling rate to 20% and the sampling duration to 45ms for the same service. This causes Wavefront to receive:
- 20% of the traces generated by that service, regardless of the length of their spans.
- Any additional spans outside of that 20% that are longer than 45ms.
As a result, the ingested sample will contain somewhat more than 20% of the generated traces, with some spans that are shorter than 45ms.
Setting Up Explicit Sampling Through the Proxy
You can set up explicit sampling strategies through a Wavefront proxy by adding the sampling properties to the proxy’s configuration file.
- On the proxy host, open the proxy configuration file
wavefront.conffor editing. The path to the file depends on the host.
- Add the traceSamplingRate property, the traceSamplingDuration property, or both to the
wavefront.conffile. In the following example, the
traceSamplingRateproperty sends 10% of the trace to Wavefront and the
traceSamplingDurationproperty sets the minimum sampling duration to 45 milliseconds:
# Number from 0.0 to 1.0 traceSamplingRate=.1 ... traceSamplingDuration=45Important: If you have more than one proxy, each proxy must have the same value for the
traceSamplingRateproperty. If different proxies send different percentages of spans to Wavefront, you get incomplete traces.
- Save the
- Start the proxy.
Setting Up Explicit Sampling in Your Code
You can set up explicit sampling strategies in application code that is built with one of the following Wavefront observability SDKs:
- The Wavefront OpenTracing SDK
- Any Wavefront observability SDK that depends on the Wavefront OpenTracing SDK
You set up a sampling strategy by configuring a Wavefront Tracer with a Sampler object. You create one Sampler for each sampling strategy. See the README file for the Wavefront observability SDK you are using.