Learn how to snooze an alert, and how to use maintenance windows to prevent alerts from firing when systems are undergoing maintenance.

You can prevent alerts from firing by using one of the following techniques:

  • To disable alert checking immediately, you can snooze an alert.
  • To disable alerts from firing for a set of sources or alerts during a custom time window, you can create a maintenance window.
  • To prevent an alert from ever firing for a set of sources, you can configure the alert condition to exclude those sources.
  • To prevent an alert from firing outside of certain hours, you can alert only between specific times.

Snoozing and Unsnoozing Alerts

You can snooze an alert, so it doesn’t fire even if the condition is met. Wavefront allows you to snooze one or more alerts for 30 minutes, 1 hour, 6 hours, 1 day, 1 week, or Forever. If you choose Forever, the alert is snoozed until it is unsnoozed.

To snooze or unsnooze one or more alerts:
  1. Click Alerting from the taskbar.
  2. Select the check boxes next to the alerts that you want to snooze.
  3. Click the Snooze drop-down menu and select the desired duration.
  4. To unsnooze the alerts, click Unsnooze.
Alert browser with alert selected and Snooze menu
To snooze or unsnooze a single alert:
  1. Click Alerting from the taskbar.
  2. Click the ellipsis icon on the left of the alert and select Snooze > <Duration>.
  3. To unsnooze the alert, click the ellipsis icon on the left of the alert and select Unsnooze.
Alert ellipsis menu with Snooze selected and snooze time options

Using Maintenance Windows

You can create maintenance windows to temporarily prevent alerts from firing when disruptive operations occur as a result of system maintenance or testing. During such operations, you know it’s likely that alerts will fire.

You can:

  • Create a maintenance window to prevent alerts from firing. You can target the maintenance window only to certain sources, alert tags, point tags, etc.
  • Close (end) maintenance windows early or make them longer.
  • Extend selected maintenance windows.
  • Send alert notifications to an alternate alert target during the maintenance window.

To view and manage maintenance windows, select Browse > Maintenance Windows.

Watch this video for an introduction to maintenance windows:

Creating a Maintenance Window

Creating a maintenance window consists of these simple steps discussed below:

  1. Specify required information, including description and start and end dates.
  2. Narrow down the scope. By default, no alerts fire during the maintenance window. You can target only specific alerts, for example, alerts for sources or environments that will be in maintenance.
  3. Optionally, specify one or more alternate alert targets. By default, no notifications are sent during the maintenance window.

Step 1: Specify Required Maintenance Window Fields

  1. Click Alerting or select Browse > Maintenance Windows from the taskbar.
  2. Click the Create Maintenance Window button.
  3. Specify the Name and Description for the maintenance window.
  4. Specify the Start Time and End Time for the maintenance window.

set maintenance window basic properties

Step 2: Narrow Down the Scope

By default, the maintenance window stops all alerts during the specified time. Usually, you’ll want to stop only alerts from a specific set of sources, or in a certain availability zone or environment. For example, you could specify a set of hosts that you expect to take down, or decide not to alert for a certain point tag (e.g. env=dev). You could also specify a set of sources with a certain source tag, for example, if you know that an availability zone will be temporarily offline.

Specify one or more of the following:

  • Alert Tags: Type one or more alert tag names to suppress any alert that has one or more specified alert tags. All alerts are included if you don't specify alert tags.
  • Point Tags: Suppress any alert that has the specified alert tags and one or more specified point tag. Example: "dev".
  • Sources: Type one or more source names to suppress any alert on a source that has a matching source. Example: "app-14"
  • Source Tags: Type one or more source tag names to suppress any alert on a source that has a matching source tag.

set maintenance window scope

Step 3 (Optional): Specify Alternate Alert Targets

By default, no alert notifications are sent during the maintenance window. The alert is mute. You can instead specify alternate alert targets to notify during the maintenance window.

set maintenance window alert notification behavior

Maintenance Window Example

Suppose you have a group of alerts that are used primarily as demo examples. These alerts have alert tag paths such as example.latency.dev, example.latency.prod, example.network.dev, example.network.prod, and so on.

To suppress the example alerts, you create a maintenance window as shown above, and fill in Affected Alerts and Sources according to your use case:

  • To suppress all of the example alerts from firing on any source:
    • In Affected Alert Tags, specify the tag path example.*.
  • To suppress just the example production alerts from firing on the source named app-1:
    • In Affected Alert Tags, specify the tag path example.*.prod.
    • In Affected Sources, specify app-1.
  • To suppress just the example latency alerts from firing either on a source that has the source tag EastCoastSources, or on the source named app-1:
    • In Affected Alert Tags, specify the tag path example.latency.*.
    • In Affected Source Tags, specify the source tag EastCoastSources.
    • In Affected Sources, specify app-1.

Extending a Maintenance Window

You can extend the duration of a maintenance window. To extend one or more maintenance windows:

  1. Select Browse > Maintenance Windows.
  2. Select the check boxes next to the maintenance windows to be extended.
  3. Click the Extend drop-down menu and select the desired duration and confirm.
extend menu with extend time choices

To extend a single maintenance window, click the ellipsis icon on the left of the window, click Extend and select the desired duration.

Closing a Maintenance Window

You can close the window before it is scheduled to finish. To close one or more maintenance windows:

  1. Select Browse > Maintenance Windows.
  2. Select the check boxes next to the maintenance windows to be closed.
  3. Click the Close button and confirm.

To close a single maintenance window, click the ellipsis icon on the left of the window and click Close.

Editing or Deleting a Maintenance Window

To edit or delete a maintenance window, click the ellipsis icon on the left of the window and click Edit or Delete.

Excluding Sources from an Alert

You can exclude sources from an alert by configuring the alert condition so that it filters out source tags that are associated with the sources to be skipped. Doing so prevents the metrics on the source from triggering the alert.

Suppose an alert condition tests the metrics that flow from sources app-1, app-2, …, app-10. You decide to decommission app-2 and replace it with a new app-11. The following steps cause the alert to filter out the metrics from the decommissioned source:

  1. Add a source tag such as decommissioned to app-2 when you are ready to take that source out of service.
  2. Modify the alert condition to include and not tag=decommissioned, for example: ts(~sample.cpu.usage.percentage, source=app-* and not tag=decommissioned) > .5.

Learn More!

  • To learn more about using point tags for maintenance windows, see this KB article