Learn how to examine and fine-tune alerts.

Alerts notify when there’s a problem, and support finding the root cause of a problem quickly. Wavefront has two GUIs:

  • Alert Viewer: When you receive an alert notification, the notification includes a link to the Alert Viewer.
    • Drill down into the alert cause (source, point tags, etc.).
    • Examine related information.
  • Alerts Browser: Allows you to investigate and manage all alerts.
    • Investigate all alerts and their state, history, and more.
    • Clone, edit, or delete one or more alerts.
    • Snooze alerts or put them in maintenance mode.

Examine an Alert in Alert Viewer

When you receive an alert notification, it includes a link to the alert in Alert Viewer. The related information in Alert Viewer can help you determine what’s going on.

annotated alert viewer allowing you to solve the problems listed below

Solve Problems with Alert Viewer

Get a 10-second briefing in the Alert description including:
  • Alert description
  • Alert settings
  • Alert targets
  • When the alert ended (if applicable)
Description of the alert
Examine Related Firing Alerts. When an alert fires, Wavefront scans all the other alerts that have fired within 30 minutes and correlates them with the initial event using AI/ML algorithms. You can filter by alert severity. Related Firing Alerts section supports filters, such as severe, warn, smoke and info.
Use the Affected section to determine what is failing.

When an alert fires, Wavefront analyzes the point tags that are most likely to be related to the firing alert and displays them in ranked order in the Alert Viewer. These point tags are a list of suspects for why the alert is firing. For example, if the alert is caused by an outage in region=us-west-2, Wavefront ranks this tag higher than other tags.
Affected point tags example
Other Firings shows past firings of the same alert with a link to the corresponding firing in the Alert Viewer. For multi-threshold alerts, you can see the severity. Click the links to see details. Other Firings list with links to the past firings
In the Data section, examine the query (or queries), filter what's displayed, and open the alert query in Chart Editor. Data section displaying the alert query and condition

The alert target mustache syntax supports a url variable and a charturl.

  • Simple notification emails include a View Alert Chart link that takes you to the chart view.
  • For PagerDuty, alert target (webhook), and templated email notifications:
    • The link target of the url mustache template variable directs to the Alert Viewer. 
    • The mustache context variable chartUrl takes you directly to the chart view. 

 

Examine and Manage All Alerts in Alerts Browser

You can view and manage all alerts from the Alerts Browser.


To examine alerts in the Alerts Browser, click Alerting in the taskbar. A colored dot next to Alerting indicates that there are firing alerts. Hover over the Alerting button in the taskbar to see how many alerts are currently firing.
multiple firing alerts on the clock icon next to text Alerting in taskbar.

To find exactly the alerts that you need you can:
  • Type the alert name in the search field
  • Use a filter, for example, select State, Severity, Services, Applications, or alert tag.
For example, you could show alerts that are both FIRING and SEVERE.
Firing and Severe selected in filter bar on left.

Examine an Alert

The Alerts Browser shows the properties and current state of an alert. For example, an alert that is firing looks like this:

Annotated screenshot highlighting the UI elements which are described in the text below

Here’s a summary of what you can do:

  • Click the ellipsis icon for a menu.
  • Click the chart icon next to the status for alert details. If the alert is firing, the Alert Viewer displays.
  • View the alert condition and points.
  • Below the severity:
    • View the last affected series, including the affected sources and point tags.
    • View the targets. For multi-threshold alerts, you see this information for each severity.
  • Examine alert tags or add a tag to make filtering for the alert easier.

View Alert Details

To view alert details, click the chart icon in the State column in the Alerts Browser.

  • If the alert is in FIRING state, the Alert Viewer displays
  • If the alert is not in FIRING state, a chart displays with these queries:

  • <Alert name> - the alert’s Display Expression, if there is one. Otherwise, the alert condition.
  • Past Firings - an events() query that shows past firings of the alert.

For example, for the Latency Dev Alert shown above, the chart looks like this:

Chart with 2 queries corresponding to alert shown in first section

View Alert History


Alert history shows the changes that have been made to an alert over time.

To access the alert history, click the ellipsis icon on the left of the alert in the Alerts Browser and click Versions.
alert history selected in menu

Alert history shows:

  • Which user made the changes.
  • The date and time the changes were made.
  • A description of the changes. You can revert back to a past alert version or clone a past alert version.

Clone or Delete an Alert

To make copies of an existing alert, then change the copy, you can clone an alert.

  1. Click Alerting in the taskbar to display the Alerts Browser.
  2. Click the ellipsis icon next to the alert.
  3. To clone the alert, select Clone, make changes when prompted, and click Save.
  4. To delete an alert, select Delete and confirm the deletion.

Edit an Alert

You can change an alert at any time.

  1. Click Alerting in the taskbar to display the Alerts Browser.
  2. Click the name of the alert you want to edit to display the Edit Alert page.
  3. Update the properties you want to change, and click Save.

You can use alert tags to organize related alerts into categories. Alert tags are especially useful for setting up maintenance windows. You can:

Manage Alert Tags


You can add a new or existing alert tag at any time:
  • Set the Tags property when you create or edit the alert.
  • Click plus (+) at the bottom of the alert in the Alerts Browser.
  • Select one or more alerts in the Alerts Browser and click +Tag or -Tag

For example, you might assign tags like networkOps, underDevelopment, and eastCoast. All users can later search for one or more of these tags to find any other alerts that are in the same category or combination of categories.

Alerts Browser, + selected for single alert, Add Existing Tag and Create New Tag options

Use Multi-Level Alert Tags

If your environment has a nested set of categories, you can use alert tag paths. For example, suppose you have created a group of alerts that you use as demo examples, and:

  • Within the demo group, some alerts monitor network activity, while others monitor request latency.
  • Within each subgroup, some alerts monitor production applications, while others monitor development applications.

To manage these alerts, you assign the tag paths example.network.prod, example.network.dev, example.latency.prod, and example.latency.dev. The Alerts Browser below shows the tag paths as a hierarchy under Tag Paths on the left. You can click example and then network to view all alerts that have a tag path that starts with example.network.

Alert tag path

When you create a maintenance window, you can use a wildcard to match tag path components:

  • example.*.* matches the entire group of demo alerts.
  • example.latency.* matches all of the alerts that monitor request latency.
  • example.*.prod matches all of the production alerts.

When you have many and complex tag paths, you can search them by parent. For example, if you have the tag paths example.network.prod, example.network.dev, example.latency.prod, and example.latency.dev, you can perform a search by example and the search returns all of its children.

Alert Events

Wavefront creates events as alerts fire, update, and resolve. You can optionally display those events as icons on a chart’s X-axis:

event icons

Backtesting

Wavefront can display actual firings or hypothetical alert-generated events using backtesting. Backtesting enables you to fine tune new or existing alert conditions before you save them.

When you create an alert, the Events Display is set to Backtesting. You can later edit the alert.

To change the events display:

  1. Select the alert and click Edit.
  2. Change the Events Display:
    • Actual Firings - Displays past alert-generated event icons on the chart. You will see how often the alert actually fired within the given chart time window.
    • Backtesting - Displays hypothetical alert-generated event icons on the chart. You can see how often an alert would fire within the chart time window based on the condition and the Alert Fires field.

Backtesting does not always exactly match the actual alert firing. For example, if data comes in late, backtest events won’t match the actual alert firing. Even if data are meeting the alert condition for the “condition is true for x mins” amount of time, the alert itself might not fire because the alert check, determined by the alert check interval, happens too soon or too late. For both cases, backtesting shows the alert as firing while the actual alert might not show as firing.

Do More!