Learn how to examine and fine-tune alerts.

Alerts help you find the root cause of a problem quickly.

When you receive an alert notification, the notification includes a link to the Alert Viewer, where you can drill down and examine related information. From the Alerts Browser, you can check and modify all alerts.

  • In Alert Viewer, view related events, point tags, and more
  • In Alerts Browser, investigate all alerts and their state, history, and more.
  • In Alerts Browser, clone, edit, or delete alerts.

Examine an Alert in Alert Viewer

When you receive an alert notification, it includes a link to the alert in Alert Viewer. The related information that Alert Viewer displays help you determine what’s going on.

annotated alert viewer

What You Can Do

Get a 10-second briefing in the Alert description including:
  • Alert description
  • Alert settings
  • Alert targets
  • When the alert ended (if applicable)
Description of the alert
Examine Related Firing Alerts. When an alert fires, Wavefront scans all the other alerts that have fired within 30 minutes and correlates them with the initial event using AI/ML algorithms. You can filter by alert severity. Related Firing Alerts section supports filters
Affected Point Tags help you determine what is failing. When an alert fires, Wavefront analyzes the point tags that are most likely to be related to the firing alert and displays them in ranked order on the Alert Viewer. These point tags become a list of suspects for why the alert is firing. For example, if the alert is caused by an outage in region=us-west-2, Wavefront ranks this tag higher than the other tags. Affected point tags example
Past Firings shows past firings with a link to the corresponding firing in the Alert Viewer. For multi-threshold alerts, you can see the severity. Data section
In the Data section you can examine the query (or queries), filter what's displayed, and open the alert query in Chart Editor. Data section

Starting with 2020.22, the alert target mustache syntax supports 2 variables:

  • The url variable
  • The charturl variable

Alert Notifications Created Before 2020.22

For any alerts that were created before 2020.14, the url variable points to the alert chart. Users who click a link in an alert notification are directed to the chart.

Alert Notifications Created in 2020.22 and later

  • Simple notification emails now include a View Alert Chart link that takes you directly to the chart view that was the link target before 2020.22.
  • For Pagerduty, alert target (webhook), and templated email notifications::
    • The link target of the url mustache template variable directs to the new Alert Viewer. 
    • A new mustache context variable chartUrl will take you directly to the chart view that was the link target before 2020.22. 

 

Examine All Alerts in the Alerts Browser

To view all alerts on your cluster, click the Alerts button to display the Alerts browser. You can use alert names or alert tags to search or filter the list of alerts. You can also filter the list by State and Severity, to view, for example, just the alerts that are both FIRING and SEVERE.

View an Alert

The Alert browser shows the properties and current state of an alert. For example, an alert that is firing looks like this:

Alert firing

Here’s a summary of what you can do:

  • Click the ellipsis (three dots) for a menu.
  • Click the chart icon next to the status for alert details. If the alert is firing, the Alert Viewer displays.
  • View the alert condition and points.
  • Below the severity:
    • View the last affected series, including the affected sources and point tags.
    • View the targets. For multi-threshold alerts, you see this information for each severity.
  • Examine alert tags or add a tag to make filtering for the alert easier.

View Alert Details

To view alert details, click the chart icon in the State column in the Alerts browser.

  • If the alert is in FIRING state, the Alert Viewer displays
  • If the alert is not in FIRING state, chart displays with these queries:

  • <Alert name> - the alert’s Display Expression, if there is one. Otherwise, the alert condition.
  • Past Firings - an events() query that shows past firings of the alert.

For example, for the alert shown above, the chart displays:

Alert queries

View Alert History

Alert history shows the changes that have been made to an alert over time. To access the alert history, click the three dots to the left of the alert in the Alerts browser and click Versions:

Alert queries

Alert history shows:

  • Which user made the changes.
  • The date and time the changes were made.
  • A description of the changes. You can revert back to or clone a past alert version.

You can also see at a glance all firing alerts from the alert icon in the task bar.

Clone or Delete an Alert

If you want to make copies of an existing alert, then change the copy slightly, you can clone the alert.

  1. Cick the Alerts button to display the Alerts page.
  2. Click the 3 dots to the left of the alert.

    Alert cloning

    • To clone an alert, click Clone, make changes when prompted, and click Save.
    • To delete an alert, click Delete and confirm the deletion.

Edit an Alert

You can change an alert at any time.

  1. Click the Alerts button to display the Alerts browser.
  2. Click the name of the alert you want to change to display the Edit Alert page.
  3. Update the properties you want to change, and click Save.

You can use alert tags to organize related alerts into categories. Alert tags let you:

You can add a new or existing alert tag at any time:

  • Set the Tags property when you create or edit the alert.
  • Click + at the bottom of the alert when you view it in the Alerts browser.

For example, you might assign tags like networkOps, underDevelopment, and eastCoast. All users can later search for one or more of these tags to find any other alerts that are in the same category or combination of categories.

Multi-Level Alert Tags

You can use alert tag paths for categories that have multiple levels. For example, suppose you have created a group of alerts that you use as demo examples, and:

  • Within the demo group, some alerts monitor network activity, while others monitor request latency.
  • Within each subgroup, some alerts monitor production applications, while others monitor development applications.

To help you manage these alerts, you assign the tag paths example.network.prod, example.network.dev, example.latency.prod, and example.latency.dev. The Alerts browser below shows the tag paths as a hierarchy under Tag Paths on the left. You can click example and then network to view all alerts that have a tag path that starts with example.network.

Alert tag path

In tasks such as creating a maintenance window, you can use a wildcard to match tag path components:

  • example.*.* matches the entire group of demo alerts.
  • example.latency.* matches all of the alerts that monitor request latency.
  • example.*.prod matches all of the production alerts.

Alert Events

As alerts fire, update, and resolve, events are created in Wavefront. You can optionally display those events as icons on a chart’s X-axis:

event icons

Backtesting

Wavefront can display actual firings or hypothetical alert-generated events using backtesting. Backtesting enables you to fine tune new or existing alert conditions before you save them.

When you create an alert, the Events Display is set to Backtesting. You can later edit the alert.

To change the events display:

  1. Select the alert and click Edit.
  2. Change the Events Display:
    • Actual Firings - Displays past alert-generated event icons on the chart. You will see how often the alert actually fired within the given chart time window.
    • Backtesting - Displays hypothetical alert-generated event icons on the chart. You can see how often an alert would fire within the chart time window based on the condition and the Alert Fires field.

Backtesting does not always exactly match the actual alert firing. For example, if data comes in late, backtest events won’t match the actual alert firing. And even if data are meeting the alert condition for the “condition is true for x mins” amount of time, the alert itself might not fire because the alert check, determined by the alert check interval, happens too soon or too late. For both cases, backtesting shows the alert as firing while the actual alert might not show as firing.

Do More!