You can view alert history and snooze and unsnooze alerts.
For additional details about how alerts work in Wavefront, see Alert States and Lifecycle.
To view and manage alerts, click the Alerts button or select Browse > Alerts.
Creating an Alert
To create an alert:
- Do one of the following:
- Alerts browser - Select Alerts and click the Create Alert button located at the top of the filter bar.
- Chart - Hover over a query field and click the Create Alert link below the query field.
The ts() expression in the selected query field populates the alert’s Condition field.
- Fill in the alert properties.
Property Description Events Display Whether to display actual or hypothetical alert firing event icons on the preview chart.
- Actual Firings (existing alerts only) - Select this radio button to display past alert-generated event icons on the chart. You will see how often the alert actually fired within the given chart time window.
- Backtesting - Select this radio button to display hypothetical alert-generated events icons on the chart. You will see how often an alert hypothetically would fire within the given chart time window based on the conditional threshold and the Alert fires field. Backtesting enables you to fine tune new or existing alert conditions before you save them.
Name Name of the alert. The name must contain 1-255 characters. Pick a simple name that makes it easy to identify the alert's purpose. Condition A conditional ts() expression that defines the threshold for the alert. You can use any valid Wavefront Query Language constructs in the expression. You can use free form query mode or the Query Builder to create the expression. The expression coupled with the Alert fires setting determines when the alert fires.
- Alert fires - Length of time during which the Condition expression must be true before the alert fires. The minimum number of minutes is 1. For example, if you enter 5, the alerting engine reviews the value of the Condition during the last 5 minute window to determine if the alert should fire or not.
- Alert resolves - Length of time during which the Condition expression must be false before the alert switches to resolved. The minimum number of minutes is 1. If you don't enable this field and specify a time, it defaults to the Alert fires setting.
Display Expression Optional. The query sent to targets when notified of alert state changes. You can use free form query mode or the Query Builder to create the expression. If not set, the query sent is the expression in the Condition field. Severity How important the alert is. In decreasing importance: SEVERE, WARN, SMOKE, and INFO. Targets Targets to notify when the alert changes state. For example, notifications are sent when an alert changes state from FIRING to CHECKING, and when an alert is snoozed. A list of: ten different email addresses, pager services such as PagerDuty and VictorOps, communication channels such as Slack and HipChat, and webhooks separated by commas. Additional Information Any additional information about the alert, such as a link to a run book. Tags Assigned alert tags. You can enter existing alert tags or create new alert tags. Property Description Checking Frequency Number of minutes between checking whether Condition is true. Minimum and default is 1. When an alert is in the INVALID state, it is checked approximately every 15 minutes, instead of the specified checking frequency. Resend Notifications Whether to resend notification of a firing alert and if enabled, the number of minutes to wait before resending the notification.
- Click Save.
Editing an Alert
To edit an alert, click the alert name in the Alerts browser or select > Edit at the far right of the alert.
Cloning and Deleting Alerts
To clone or delete an alert, select > [Clone | Delete] at the far right of the alert.
To delete one or more alerts, select the checkboxes next to one or more alerts and click .
Exploring Alert History
Alert history provides you with changes that have been made to an alert over time. You can access the alert history by selecting > Versions from the menu located to the right of an alert on the Alerts page. When you select Versions, a page displays contain a list of versions of the alert. Alert history tells you which user made the changes, the date and time the changes were made, and a description of the changes. You can revert back to or clone a past alert version. Alert history was implemented in Q4 of 2015, so you may not see any change history prior to that time if the alert was created before that time.
Snoozing and Unsnoozing Alerts
There are certain times when you want to silence an alert, whether the conditional is met or not. You can do this by snoozing an alert. Wavefront allows you to snooze one or more alerts for 30 minutes, 1 hour, 6 hours, 1 day, 1 week, or Forever. If you choose Forever, the alert is snoozed until it is unsnoozed.
To snooze one or more alerts:
- Check the checkboxes next to the desired alert(s).
- Click the Snooze dropdown and select the desired duration.
- Click the Snooze confirmation.
To snooze a single alert, select Snooze > <Duration> at the far right of the alert.
To unsnooze alerts, check the checkboxes next to the alerts and select Snooze > Unsnooze. To unsnooze a single alert, select Snooze > Unsnooze at the far right of the alert.
Managing Alert Tags
See Organizing with Tags.