Learn control trace data sampling with sampling policies.

VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront) uses intelligent sampling to reduce the volume of ingested traces. The goal of intelligent sampling is to retain traces that are likely to be informative. But sometimes intelligent sampling discards traces that you want to keep. You can create a sampling policy to fine-tune intelligent sampling and indicate that you want to keep certain spans. Creating a sampling policy affects your costs as you store more data.

You can create a sampling policy, edit, delete, restore, deactivate, and see the version history of the policy you created:

This diagrams shows the user flow from creating a policy, to edititng, deleting, restoring, deactivation, and seeing teh version history of a policy.

Let’s look at how you can create a sampling policy and get familiar with the policy expressions.

Create a Sampling Policy

To create a sampling policy:

  1. Log in to your service instance.
  2. From the toolbar, select Applications > Sampling Policies.
  3. Click Create Policy and enter the details:
    UI Element Description
    Policy Name The name of the policy. The policy name is unique, and you cannot create another policy with the same name.
    Description Add a description to explain what the policy does. It helps administrators understand why the policy is in place.
    Policy Expression Define a policy expression indicate which spans you want to keep. This expression needs to be in the YAML format. See Add a Policy Expression for details.
    Sampling percentage Define the percentage of spans you want to retain by entering a value between 0 and 100. For example, 80 indicates that you want to keep 80% of the spans that meet the policy expression.
    Status The policy is in effect when it is in the active state. Once the policy is active, it affects your costs as you store more data.
    To see the number of spans stored per second after a sampling policy is created, see Track the Volume of Stored Trace Data.
  4. Click Create Policy. the screenshot shows the create policy form with all the UI elements that were described in the table above.

Now, you see the policy you created listed under sampling policies.

Add a Policy Expression

When you create a sampling policy, you need to define a policy expression that specifies the spans you want to keep. This expression needs to be in the YAML format.

Let’s look at a policy expression that asks the Operations for Applications service to store traces if the application name is beachshirts.

Attribute Description
Key You can pass the following values and any span tag key as a key:
  • If a span tag key or a default key is used to pass string values, the key needs to be defined between {{}}.
    Example:
    • {{spanName}}
    • {{sourceName}}
  • If a span tag key or a default key is used to pass an integer value, the key needs to start with a $.
    Example:
    • $startMillis
    • $duration in milliseconds.
Value Define the values as follows:
  • For string values, surround the value with double quotes ("").
    Example: Retain spans that have the application name set to beachshirts. 
    {{application}} = "beachshirts"
  • For integer values, you don't need to use double quotes.
    Example: Retain spans that take more than 1000 milliseconds to complete. 
    $duration>1000
Operation Policy expressions can contain operations that are cases sensitive, and operations that are not case sensitive:

Operations that are case sensitive: These operations match only if the case is the same. For example, {{application}} = "beachshirts" is equal to beachshirts but not beachShirts.
  • =, >, <, <=, >=, !=
  • equals, startsWith, contains, endsWith, matches
  • and, or
  • not
  • in
Operations that are not case sensitive: These operations match even if the case is not the same.
  • equalsIgnoreCase
  • startsWithIgnoreCase
  • endsWithIgnoreCase
  • containsIgnoreCase
  • matchesIgnoreCase

Advanced Example

The following policy expression creates a sampling policy to keep all spans from the beachshirts application where the operation starts with Shopping, the source contains prod, and the status code is 400 or 404.

{{application}} = "beachshirts" and
{{spanName}} startsWith "Shopping." and
{{sourceName}} contains "prod" and
{{http.status_code}} in ("400", "404")

Edit a Sampling Policy

To edit a sampling policy:

  1. Click the vertical ellipsis next to the policy name and click Edit. the screenshot described above, with the edit option highlighted with a red box.

  2. Update the details, and click Update Policy.

Delete a Sampling Policy

To delete a sampling policy, click the vertical ellipsis next to the policy name and click Delete.

To see the policies you deleted, click View Deleted Policies.

Restore a Sampling Policy

To restore a deleted policy:

  1. Click View Deleted Policies.
  2. Click the vertical ellipsis next to the policy name and click Restore.

Deactivate or Activate Sampling Policy

Deactivate a Sampling Policy

To deactivate a sampling policy, Click the vertical ellipsis next to the policy name and click Deactivate.

When the policy is deactivated, it changes to an inactive state. By default, you don’t see inactive policies. Enable the Show Inactive toggle to see the deactivate policies in the sampling policies list.

Example: Shows the how inactive sampling policies toggle is enabled and as a result how you see the inactive sampling policy in the list of policies.

Activate a Sampling Policy

To activate a sampling policy:

  1. Enable the Show Inactive toggle to see the deactivate policies.
  2. Click the vertical ellipsis next to the inactive policy name, and click Activate.

Learn More

To learn more about our trace data sampling strategies, see Trace Sampling.