Learn how alerts work, and how to create and examine them.

With Wavefront, you can create smart alerts that dynamically filter noise and capture true anomalies. When you create an alert, you specify one or more alert targets that receive the alert notification(s). You can view an image of the chart in the alert notification, and click a link to see the alert in context. The end result is fewer false alerts and faster remediation when real issues occur.

Wavefront Alerts

An alert defines:

  • The condition under which metric values indicate a system problem.
  • One or more targets to notify when the condition evaluates to true or false for a specified period of time.
  • Optionally, information about the alert notification format.

Wavefront supports classic alerts, where each alert has one preset severity, and multi-threshold alerts, where an alert can have different severities for different threshold values.

Alert Condition

The alert condition is a ts() expression that defines the threshold for an alert.

  • If an alert’s Condition field is set to a conditional expression, for example ts("requests.latency") > 195, then all data values that satisfy the condition are marked as true (1) and all data values that do not satisfy the condition are marked as false (0).
  • If the Condition field is set to a ts() expression, for example ts("cpu.loadavg.1m"), then all non-zero data values are marked as true and all zero data values are marked as false. If there is no reported data, then values are neither true nor false.

An alert fires when a metric stays at a value that indicates a problem for the specified amount of time.

  • A classic alert send a notification with the specified severity to all specified targets.
  • A multi-threshold alert allows you to specify multiple severities and a different target for each severity.

Alert Target

Each alert is associated with one or more alert targets. The alert target specifies who to notify when the alert changes state.

  • For classic alerts, you specify a severity and one or more corresponding alert targets. You can set up email, PagerDuty, and custom alert targets.
  • For multi-threshold alerts, you can specify a different alert target for each threshold, for example, an email target when the alert reaches the INFO threshold and a PagerDuty target when the alert reaches the SEVERE threshold. You can specify only custom alert targets, but it’s easy to set up a custom email or PagerDuty alert target.

Alert Basics Videos

In this video, Clement explains how classic alerts work:

In this video, Jason explains classic alerts while he’s showing them in the UI:

Creating an Alert

You can create a classic alert with a single severity level (e.g. SEVERE) or a multi-threshold alert, which allows you to customize alert behavior for different thresholds. For each threshold, you select a corresponding severity and one or more targets to notify in case the threshold is met.

Creating a Classic Alert

Required fields for a classic alert are:

  • Alert name (default is New Alert)
  • Alert condition
  • Alert severity

You also specify one or more alert targets to notify when the alert fires.

To create a classic alert:

  1. Do one of the following:
    • Alerts browser - Select Alerts and click the Create Alert button located at the top of the filter bar.
    • Chart - Hover over a query field and click the Create Alert link below the query field. create_alert
  2. Specify the following required alert properties.
    PropertyDescription
    Name Name of the alert. 1-255 characters.
    Condition A conditional ts() expression that defines the threshold for the alert. The condition expression can include any valid Wavefront Query Language construct. The condition expression coupled with the Alert fires setting determines when the alert fires.
    • Alert fires - Length of time (in minutes) during which the Condition expression must be true before the alert fires. Minimum is 1. For example, if you enter 5, the alerting engine reviews the value of the condition during the last 5 minute window to determine whether the alert should fire.
    • Alert resolves - Length of time (in minutes) during which the Condition expression must be not true before the alert switches to resolved. Minimum is 1. Omit this setting or pick a value that is greater than or equal to the Alert fires value to avoid resolve-fire cycles.
    For details and examples, see Alert States and Lifecycle.
    Severity How important the alert is. In decreasing importance: SEVERE, WARN, SMOKE, and INFO.
  3. (Recommended) Specify a list of alert targets to notify when the alert changes state, for example, from CHECKING to FIRING, or when the alert is snoozed. You can specify up to ten different targets across the following types. Use commas to separate targets of the same type.
    PropertyDescription
    Email Valid email addresses. Alert notifications are sent to these addresses in response to a default set of triggering events, and contain default HTML-formatted content.
    PagerDuty Key PagerDuty keys obtained by following the steps for the PagerDuty integration. Alert notifications that use these keys are sent in response to a default set of triggering events, and contain default content.
    Alert Target Names of custom alert targets that you have previously created to:
    • Configure webhook notifications for pager services and communication channels. Follow the steps for the VictorOps integration, Slack integration, or HipChat integration for notifications on these popular messaging platforms.
    • Configure email or PagerDuty notifications with nondefault content or triggers.
  4. (Recommended) Specify a Display Expression. Defaults to the value of the condition expression, either 0 or 1. Specify a display expression to get more details when the alert changes state. The display expression can include any valid Wavefront Query Language construct, and typically captures the underlying time series that the condition expression is testing. The results of the display expression are:
    • Shown in the Events Display preview chart on the page for creating or editing the alert.
    • Shown in any chart image that is included in a notification triggered by the alert.
    • Shown in the interactive chart you can visit from a notification triggered by the alert.
    • Used as the basis for any statistics that you might include in a custom notification triggered by the alert.
  5. (Optional) To help you find the alert and information about it in the Alerts browser, specify Additional Information and Tags.
    PropertyDescription
    Additional Information Any additional information, such as a link to a run book.
    Tags Tags assigned to the alert. You can enter existing alert tags or create new alert tags. See Organizing with Tags.
  6. (Optional) Click the Advanced link to configure the following alert properties. The defaults for those properties are often appropriate.
    PropertyDescription
    Checking Frequency Number of minutes between checking whether Condition is true. Minimum and default is 1. When an alert is in the INVALID state, it is checked approximately every 15 minutes, instead of the specified checking frequency.
    Resend Notifications Whether to resend notification of a firing alert. If enabled, you can specify the number of minutes to wait before resending the notification.
    Metrics Whether to include obsolete metrics. If enabled, the alert considers metrics that have not reported for 4 weeks or more. Customers who use queries that aggregate data in longer time frames sometimes want to include those older metrics.
  7. Click Save.

Video: Creating a Classic Alert

This video shows how Jason creates a classic alert:

Creating a Multi-Threshold Alert

Required fields for a multi-threshold alert are:

  • Alert name (defaults to New Alert)
  • Alert condition and operator (e.g. greater than>)
  • At least one severity and corresponding threshold value. For each severity, you can specify one or more alert targets to notify when the alert changes state. Only custom alert targets are supported, but you can initially create the alert without specifying a target.

In contrast to classic alerts, Wavefront creates a display expression for a multi-threshold alert. The expression shows the alert condition.

To create a multi-threshold alert:

  1. Do one of the following:
    • Alerts browser - Click the Alerts button, then click the Create Alert button located at the top of the filter bar.
    • Chart - Hover over a query field and click the Create Alert link below the query field. create_alert
  2. Next to Type, click Threshold.
  3. Fill in the following required alert properties.
    PropertyDescription
    Name Name of the alert. 1-255 characters.
    Condition A ts() expression that defines the threshold for the alert. The condition expression can include any valid Wavefront Query Language construct. The condition expression coupled with the Alert fires setting determines when the alert fires.
    • Alert fires - Length of time (in minutes) during which the Condition expression must be true before the alert fires. Minimum is 1. For example, if you enter 5, the alerting engine reviews the value of the condition during the last 5 minute window to determine whether the alert should fire.
    • Alert resolves - Length of time (in minutes) during which the Condition expression must be not true before the alert switches to resolved. Minimum is 1. Omit this setting or pick a value that is greater than or equal to the Alert fires to avoid potential chains of resolve-fire cycles.
    For details and examples, see Alert States and Lifecycle.
    Operator Select one of the operators, for example, greater than or . The operator determines which values are allowed for the different severity thresholds. For example, if the operator is greater than, then SEVERE must be the highest number and INFO must be the lowest number, and the numbers must increase from INFO to SEVERE. You don't have to specify all 4 severities.
    Severity For multi-threshold alerts, specify more than one severity - or create a Classic alert. Associate a threshold value with each severity. The order must match the operator. For example, you can specify a Operator =>, SEVERE 6000, and WARN 5000, but you can't specify SEVERE 5000, and WARN 6000 with that operator.
  4. (Recommended) Specify a list of alert targets for each severity. Wavefront notifies the target(s) when the alert changes state, for example, from CHECKING to FIRING, or when the alert is snoozed. You can specify up to ten different targets for each severity, but you can use each target only for one severity. Use commas to separate targets. For multi-threshold alerts, you have to specify names of custom alert targets that you already created. You cannot specify an email address or PagerDuty key.
  5. (Optional) To help you find the alert and information about it, specify Additional Information and Tags.
    PropertyDescription
    Additional Information Any additional information, such as a link to a run book.
    Tags Tags assigned to the alert. You can enter existing alert tags or create new alert tags. See Organizing with Tags.
  6. (Optional) Click the Advanced link to configure the following alert properties. The defaults for those properties are often appropriate.
    PropertyDescription
    Checking Frequency Number of minutes between checking whether Condition is true. Minimum and default is 1. When an alert is in the INVALID state, it is checked approximately every 15 minutes, instead of the specified checking frequency.
    Resend Notifications Whether to resend notification of a firing alert. If enabled, you can specify the number of minutes to wait before resending the notification.
    Metrics Whether to include obsolete metrics. If enabled, the alert considers metrics that have not reports for 4 weeks or more. Customers who use queries that aggregate data in longer timeframes sometimes want to include those older metrics.
  7. Click Save.

Video: Creating a Multi-Threshold Alert

This video shows how to create a multi-threshold alert:

threshold alerts

Cloning or Deleting an Alert

If you want to make copies of an existing alert, then change the copy slightly, you can clone the alert.

  1. Cick the Alerts button to display the Alerts page.
  2. Click the 3 dots to the left of the alert.

    Alert cloning

    • To clone an alert, click Clone, make changes when prompted, and click Save.
    • To delete an alert, click Delete and confirm the deletion.

Viewing Alerts and Alert History

To view alerts, click the Alerts button. A list of alerts displays. Here’s an example that shows when the alert fires that is described in Tutorial: Getting Started:

Alert firing

To view alert details, click the chart icon in the State column. A chart displays with two queries:

  • <Alert name> - the alert condition.
  • Past Firings - an events() query that shows past firings of the alert.

For example, for the alert shown above, the chart displays:

Alert queries

The Firings column shows how many times an alert changed from non-firing to firing in the last day, week, and month.

Alert history shows the changes that have been made to an alert over time. To access the alert history, click the three dots to the left of the alert on the Alerts page and click Versions. Alert history shows:

  • Which user made the changes.
  • The date and time the changes were made.
  • A description of the changes. You can revert back to or clone a past alert version.

You can also see at a glance all firing alerts from the alert icon in the task bar.

Editing an Alert

You can change an alert at any time.

  1. Click the Alerts button to display the Alerts page.
  2. Click the name of the alert you want to change to display the Edit Alert page.
  3. Update the properties you want to change, and click Save.

Alert Events

As alerts fire, update, and resolve, events are created in Wavefront. You can optionally display those events as icons on a chart’s X-axis:

event icons

Backtesting

Wavefront can display actual firings or hypothetical alert-generated events using backtesting. Backtesting enables you to fine tune new or existing alert conditions before you save them.

When you create an alert, the Events Display is set to Backtesting. You can later edit the alert.

To change the events display:

  1. Select the alert and click Edit.
  2. Change the Events Display:
    • Actual Firings - Displays past alert-generated event icons on the chart. You will see how often the alert actually fired within the given chart time window.
    • Backtesting - Displays hypothetical alert-generated event icons on the chart. You can see how often an alert would fire within the chart time window based on the condition and the Alert Fires field.

Backtesting does not always exactly match the actual alert firing. For example, if data comes in late, backtest events won’t match the actual alert firing. And even if data are meeting the alert condition for the “condition is true for x mins” amount of time, the alert itself might not fire because the alert check, determined by the alert check interval, happens too soon or too late. For both cases, backtesting shows the alert as firing while the actual alert might not show as firing.