Learn how snooze an alert, and how to use maintenance windows to prevent alerts from firing when systems are undergoing maintenance.

You can prevent alerts from firing by using one of the following techniques:

  • To disable alert checking immediately, you can snooze an alert.
  • To disable alerts from firing for a set of sources or alerts during a custom time window, you can create a maintenance window.
  • To prevent an alert from ever firing for a set of sources, you can configure the alert condition to exclude those sources.
  • To prevent an alert from firing outside of certain hours, you can alert only between specific times.

Snoozing and Unsnoozing Alerts

You can snooze an alert so it doesn’t fire even if the condition is met. Wavefront allows you to snooze one or more alerts for 30 minutes, 1 hour, 6 hours, 1 day, 1 week, or Forever. If you choose Forever, the alert is snoozed until it is unsnoozed.

To snooze one or more alerts:

  1. Check the check boxes next to the desired alert(s).
    • To snooze the alerts, click the Snooze dropdown, select the desired duration, and click OK.
    • To unsnooze the alerts, click Snooze > Unsnooze.

To snooze or unsnooze a single alert:

  • Select Snooze > <Duration> at the far left of the alert to snooze it.
  • Select Snooze > Unsnooze at the far left of the alert to unsnooze it.

Using Maintenance Windows

A maintenance window defines a time window when disruptive operations occur as a result of system maintenance or testing. During such operations, it’s likely that alerts will fire. You can create a maintenance window to prevent alerts from firing.

You can close (end) maintenance windows early or you can make them longer.

To view and manage maintenance windows, select Browse > Maintenance Windows.

Watch this video for an introduction to maintenance windows:

Creating a Maintenance Window

To create a maintenance window:

  1. Click Alerts or Browse > Maintenance Windows.
  2. Click the Create Maintenance Window button at the top of the filter bar.
  3. Fill in the maintenance window properties:

    PropertyDescription
    Name The name of the maintenance window.
    Description Additional information about the maintenance window. Information entered into this field appears directly below the maintenance window in the Maintenance Windows browser.
    Start Time The start time of the maintenance window:
    • Now - The maintenance window starts immediately.
    • - The maintenance window starts on the specified date and time. Click the text field and choose a date and time or type a date and time in the format MM/DD/YYYY HH:MM [AM|PM].
    End Time The end time of the maintenance window. The end time must be after the start time. Click the text field and choose a date and time or type a date and time in the format MM/DD/YYYY HH:MM [AM|PM].
    Affected Alerts and Sources The alerts to be suppressed during the maintenance window. You must specify at least one alert tag, source, or source tag.
    • Specify one or more alert tags or tag paths in the Affected Alert Tags field to suppress any alert that has a matching alert tag.
    • Specify one or more source tags or tag paths in the Affected Source Tags field to suppress any alert that would have met its alert condition on a source that has a matching source tag.
    • Specify one or more source names in the Affected Sources field to suppress any alert that would have met its alert condition on a matching source.
    You can omit alert tags to prevent any alert from firing on a specified source. You can combine alert tags with source names and/or source tags to prevent any alert with a specified tag from firing on a specified source.
  4. Click Save.

Example

Suppose you have a group of alerts that are used primarily as demo examples. These alerts have alert tag paths like example.latency.dev, example.latency.prod, example.network.dev, example.network.prod, and so on.

To suppress the example alerts, you create a maintenance window as shown above, and fill in Affected Alerts and Sources according to your use case:

  • To suppress all of the example alerts from firing on any source:
    • In Affected Alert Tags, specify the tag path example.*.
  • To suppress just the example production alerts from firing on the source named app-1:
    • In Affected Alert Tags, specify the tag path example.*.prod.
    • In Affected Sources, specify app-1.
  • To suppress just the example latency alerts from firing either on a source that has the source tag EastCoastSources or on the source named app-1:
    • In Affected Alert Tags, specify the tag path example.latency.*.
    • In Affected Source Tags, specify the source tag EastCoastSources.
    • In Affected Sources, specify app-1.
    • Note: If you wanted to suppress the alerts from firing on app-1 only if that source also has the source tag EastCoastSources, you can click on OR and select AND.

Extending a Maintenance Window

You can extend the duration of a maintenance window. To extend one or more maintenance windows:

  1. Select Browse > Maintenance Windows.
  2. Check the checkboxes next to the maintenance windows to be extended.
  3. Click the Extend dropdown and select the desired duration.
  4. Click the confirmation.

To extend a single maintenance window, select the three dots on the left, click Extend and select the desired duration.

Closing a Maintenance Window

You can close the window before it is scheduled to finish. To close one or more maintenance windows:

  1. Select Browse > Maintenance Windows.
  2. Check the checkboxes next to the maintenance windows to be closed.
  3. Click the Close button.
  4. Click the confirmation.

To close a single maintenance window, select the three dots on the left, and click Close.

Editing or Deleting a Maintenance Window

To edit or delete a maintenance window, select the three dots on the left and click Edit or Delete.

Excluding Sources from an Alert

You can exclude sources from an alert by configuring the alert condition so that it filters out source tags that are associated with the sources to be skipped. Doing so prevents the metrics on the source from triggering the alert.

Suppose an alert condition tests the metrics that flow from sources app-1, app-2, …, app-10. You decide to decommission app-2 and replace it with a new app-11. The following steps cause the alert to filter out the metrics from the decommissioned source:

  1. Add a source tag such as decommissioned to app-2 when you are ready to take that source out of service.
  2. Modify the alert condition to include and not tag=decommissioned, for example: ts(~sample.cpu.usage.percentage, source=app-* and not tag=decommissioned) > .5