Alerts help you find the root cause of a problem quickly.
When you receive an alert notification, the notification includes a link to the Alert Viewer, where you can drill down and examine related information. From the Alerts Browser, you can check and modify all alerts.
- In Alert Viewer, view related events, point tags, and more
- In Alerts Browser, investigate all alerts and their state, history, and more.
- In Alerts Browser, clone, edit, or delete alerts.
Examine an Alert in Alert Viewer
When you receive an alert notification, it includes a link to the alert in Alert Viewer. The related information that Alert Viewer displays help you determine what’s going on.
What You Can Do
Get a 10-second briefing in the Alert description including:
|Examine Related Firing Alerts. When an alert fires, Wavefront scans all the other alerts that have fired within 30 minutes and correlates them with the initial event using AI/ML algorithms. You can filter by alert severity.|
|Affected Point Tags help you determine what is failing. When an alert fires, Wavefront analyzes the point tags that are most likely to be related to the firing alert and displays them in ranked order on the Alert Viewer. These point tags become a list of suspects for why the alert is firing. For example, if the alert is caused by an outage in region=us-west-2, Wavefront ranks this tag higher than the other tags.|
|Past Firings shows past firings with a link to the corresponding firing in the Alert Viewer. For multi-threshold alerts, you can see the severity.|
|In the Data section you can examine the query (or queries), filter what's displayed, and open the alert query in Chart Editor.|
How Alert Notifications Include Links
Starting with 2020.22, the alert target mustache syntax supports 2 variables:
Alert Notifications Created Before 2020.22
For any alerts that were created before 2020.14, the
url variable points to the alert chart. Users who click a link in an alert notification are directed to the chart.
Alert Notifications Created in 2020.22 and later
- Simple notification emails now include a View Alert Chart link that takes you directly to the chart view that was the link target before 2020.22.
- For Pagerduty, alert target (webhook), and templated email notifications::
- The link target of the
urlmustache template variable directs to the new Alert Viewer.
- A new mustache context variable
chartUrlwill take you directly to the chart view that was the link target before 2020.22.
- The link target of the
chartUrl. To give users the option to view the chart in Chart Editor, edit existing alert targets.
Examine All Alerts in the Alerts Browser
To view all alerts on your cluster, click the Alerts button to display the Alerts browser. You can use alert names or alert tags to search or filter the list of alerts. You can also filter the list by State and Severity, to view, for example, just the alerts that are both FIRING and SEVERE.
View an Alert
The Alert browser shows the properties and current state of an alert. For example, an alert that is firing looks like this:
Here’s a summary of what you can do:
- Click the ellipsis (three dots) for a menu.
- Click the chart icon next to the status for alert details. If the alert is firing, the Alert Viewer displays.
- View the alert condition and points.
- Below the severity:
- View the last affected series, including the affected sources and point tags.
- View the targets. For multi-threshold alerts, you see this information for each severity.
- Examine alert tags or add a tag to make filtering for the alert easier.
View Alert Details
To view alert details, click the chart icon in the State column in the Alerts browser.
- If the alert is in FIRING state, the Alert Viewer displays
If the alert is not in FIRING state, chart displays with these queries:
- <Alert name> - the alert’s Display Expression, if there is one. Otherwise, the alert condition.
- Past Firings - an events() query that shows past firings of the alert.
For example, for the alert shown above, the chart displays:
View Alert History
Alert history shows the changes that have been made to an alert over time. To access the alert history, click the three dots to the left of the alert in the Alerts browser and click Versions:
Alert history shows:
- Which user made the changes.
- The date and time the changes were made.
- A description of the changes. You can revert back to or clone a past alert version.
You can also see at a glance all firing alerts from the alert icon in the task bar.
Clone or Delete an Alert
If you want to make copies of an existing alert, then change the copy slightly, you can clone the alert.
- Cick the Alerts button to display the Alerts page.
Click the 3 dots to the left of the alert.
- To clone an alert, click Clone, make changes when prompted, and click Save.
- To delete an alert, click Delete and confirm the deletion.
Edit an Alert
You can change an alert at any time.
- Click the Alerts button to display the Alerts browser.
- Click the name of the alert you want to change to display the Edit Alert page.
- Update the properties you want to change, and click Save.
Organize Related Alerts With Alert Tags
You can use alert tags to organize related alerts into categories. Alert tags let you:
- Search or filter the list of alerts in the Alerts browser to show only a category of alerts.
- Suppress a category of alerts during a maintenance window.
- Reference a group of alert metrics in a single expression.
You can add a new or existing alert tag at any time:
- Set the Tags property when you create or edit the alert.
- Click + at the bottom of the alert when you view it in the Alerts browser.
For example, you might assign tags like
eastCoast. All users can later search for one or more of these tags to find any other alerts that are in the same category or combination of categories.
Multi-Level Alert Tags
You can use alert tag paths for categories that have multiple levels. For example, suppose you have created a group of alerts that you use as demo examples, and:
- Within the demo group, some alerts monitor network activity, while others monitor request latency.
- Within each subgroup, some alerts monitor production applications, while others monitor development applications.
To help you manage these alerts, you assign the tag paths
example.latency.dev. The Alerts browser below shows the tag paths as a hierarchy under Tag Paths on the left. You can click example and then network to view all alerts that have a tag path that starts with
In tasks such as creating a maintenance window, you can use a wildcard to match tag path components:
example.*.*matches the entire group of demo alerts.
example.latency.*matches all of the alerts that monitor request latency.
example.*.prodmatches all of the production alerts.
Wavefront can display actual firings or hypothetical alert-generated events using backtesting. Backtesting enables you to fine tune new or existing alert conditions before you save them.
When you create an alert, the Events Display is set to Backtesting. You can later edit the alert.
To change the events display:
- Select the alert and click Edit.
- Change the Events Display:
- Actual Firings - Displays past alert-generated event icons on the chart. You will see how often the alert actually fired within the given chart time window.
- Backtesting - Displays hypothetical alert-generated event icons on the chart. You can see how often an alert would fire within the chart time window based on the condition and the Alert Fires field.
Backtesting does not always exactly match the actual alert firing. For example, if data comes in late, backtest events won’t match the actual alert firing. And even if data are meeting the alert condition for the “condition is true for x mins” amount of time, the alert itself might not fire because the alert check, determined by the alert check interval, happens too soon or too late. For both cases, backtesting shows the alert as firing while the actual alert might not show as firing.