Learn how to examine and fine-tune alerts.

Alerts help you find the root cause of a problem quickly.

When you receive an alert notification, the notification includes a link to the alert chart, where you can. From the Alerts Browser, you can check and modify all alerts.

Examine All Alerts in the Alerts Browser

To view all alerts on your cluster, click the Alerts button to display the Alerts browser. You can use alert names or alert tags to search or filter the list of alerts. You can also filter the list by State and Severity, to view, for example, just the alerts that are both FIRING and SEVERE.

View an Alert

The Alert browser shows the properties and current state of an alert. For example, an alert that is firing looks like this:

Alert firing

View Alert Details

To view alert details, click the chart icon in the State column in the Alerts browser. A chart displays with these queries:

  • <Alert name> - the alert’s Display Expression, if there is one. Otherwise, the alert condition.
  • Past Firings - an events() query that shows past firings of the alert.

For example, for the alert shown above, the chart displays:

Alert queries

View Alert History

Alert history shows the changes that have been made to an alert over time. To access the alert history, click the three dots to the left of the alert in the Alerts browser and click Versions:

Alert queries

Alert history shows:

  • Which user made the changes.
  • The date and time the changes were made.
  • A description of the changes. You can revert back to or clone a past alert version.

You can also see at a glance all firing alerts from the alert icon in the task bar.

Clone or Delete an Alert

If you want to make copies of an existing alert, then change the copy slightly, you can clone the alert.

  1. Cick the Alerts button to display the Alerts page.
  2. Click the 3 dots to the left of the alert.

    Alert cloning

    • To clone an alert, click Clone, make changes when prompted, and click Save.
    • To delete an alert, click Delete and confirm the deletion.

Edit an Alert

You can change an alert at any time.

  1. Click the Alerts button to display the Alerts browser.
  2. Click the name of the alert you want to change to display the Edit Alert page.
  3. Update the properties you want to change, and click Save.

You can use alert tags to organize related alerts into categories. Alert tags let you:

You can add a new or existing alert tag at any time:

  • Set the Tags property when you create or edit the alert.
  • Click + at the bottom of the alert when you view it in the Alerts browser.

For example, you might assign tags like networkOps, underDevelopment, and eastCoast. All users can later search for one or more of these tags to find any other alerts that are in the same category or combination of categories.

Multi-Level Alert Tags

You can use alert tag paths for categories that have multiple levels. For example, suppose you have created a group of alerts that you use as demo examples, and:

  • Within the demo group, some alerts monitor network activity, while others monitor request latency.
  • Within each subgroup, some alerts monitor production applications, while others monitor development applications.

To help you manage these alerts, you assign the tag paths example.network.prod, example.network.dev, example.latency.prod, and example.latency.dev. The Alerts browser below shows the tag paths as a hierarchy under Tag Paths on the left. You can click example and then network to view all alerts that have a tag path that starts with example.network.

Alert tag path

In tasks such as creating a maintenance window, you can use a wildcard to match tag path components:

  • example.*.* matches the entire group of demo alerts.
  • example.latency.* matches all of the alerts that monitor request latency.
  • example.*.prod matches all of the production alerts.

Alert Events

As alerts fire, update, and resolve, events are created in Wavefront. You can optionally display those events as icons on a chart’s X-axis:

event icons

Backtesting

Wavefront can display actual firings or hypothetical alert-generated events using backtesting. Backtesting enables you to fine tune new or existing alert conditions before you save them.

When you create an alert, the Events Display is set to Backtesting. You can later edit the alert.

To change the events display:

  1. Select the alert and click Edit.
  2. Change the Events Display:
    • Actual Firings - Displays past alert-generated event icons on the chart. You will see how often the alert actually fired within the given chart time window.
    • Backtesting - Displays hypothetical alert-generated event icons on the chart. You can see how often an alert would fire within the chart time window based on the condition and the Alert Fires field.

Backtesting does not always exactly match the actual alert firing. For example, if data comes in late, backtest events won’t match the actual alert firing. And even if data are meeting the alert condition for the “condition is true for x mins” amount of time, the alert itself might not fire because the alert check, determined by the alert check interval, happens too soon or too late. For both cases, backtesting shows the alert as firing while the actual alert might not show as firing.

Do More!