You can create custom alert targets to configure alert notifications for a variety of messaging platforms (email, pager services) and communication channels. You can route notifications for the same alert to different targets based on a point tag.
Learn About Alert Targets
This page explains how to create and manage a custom alert target.
You can further customize the contents of the alert notifications using Moustache syntax.
Our blog post Engineering Tips Series: How Wavefront’s Devops Team Uses Alert Targets to Provide Exceptional Quality of Services to Customers explains how alert targets help Wavefront to keep things running smoothly.
For the following integrations, you can follow the steps in the integration. Log in to your Wavefront instance or look at the following pages:
Why Alert Targets?
Alert targets specify when and how to send notifications.
During alert creation, you can specify an email address or a PagerDuty key in the target list. You implicitly use built-in Wavefront alert targets. These simple alert targets:
- Cause notifications to be sent whenever the alert is firing, updated, resolved, snoozed, or in a maintenance window.
- Provide internal templates for the notification contents. These internal templates are maintained by Wavefront, and may change from release to release.
For more flexibility, you can create custom alert targets that specify.
- Where to send notifications
- What kind of information
- Notification format
- Which alert events should trigger notifications.
For example, you could use a custom alert target to:
- Expand (or limit) the set of triggering events for notifications.
- Configure different contents for notifications triggered by different events.
- Associate a short name with a long list of email addresses or a lengthy PagerDuty key.
View Custom Alert Targets
To view alert targets, select Browse > Alert Targets.
Create a Custom Alert Target
The process for creating an alert target is similar for the different types of targets. Setting the Type changes which fields are displayed.
- Select Browse > Alert Targets.
- Click the Create Alert Target button.
- Fill in the properties that are common to all alert targets.
Property Description Name Name of the alert target. Pick a name that is simple and that makes it easy to identify the alert target's purpose. Description Description of the alert target. Triggers One or more alert state changes that trigger the alert target. The options are:
- Alert Firing - Trigger when the alert transitions from checking to firing.
- Alert Status Updated - Trigger when at least one time series changes category while the alert continues firing. For example, an individual time series could start to fail (satisfy the alert condition during the Alert fires time window) or could recover (stop satisfying the alert condition during the Alert resolves time window).
- Alert Resolved - Trigger when the alert resolves.
- Alert Affected by Maintenance Window - Trigger when a firing alert is affected by a maintenance window.
- Alert Snoozed - Trigger when the alert is snoozed.
- Alert Has No Data - Trigger when the time series associated with the alert have all stopped reporting data.
- Alert Has No Data Resolved - Trigger when at least one time series associated with the alert has started reporting data, while all other time series are still reporting no data.
- Alert Entered Maintenance From No Data - Trigger when none of the alert's time series are reporting data, and the alert is affected by a maintenance window.
- From the Type pull-down menu, select the alert target type:
Type Description Webhook Alert target for sending notifications to messaging platforms such as Slack, VictorOps, or HipChat. This alert target defines the HTTP callback (POST request and URL) that is triggered when an alert changes state. Alert target for sending notifications to email systems. This alert target specifies the attributes of the email messages to be sent when an alert changes state. PagerDuty Alert target for sending notifications to PagerDuty. This alert target specifies the PagerDuty key and a POST body to use when an alert changes state.
- Fill in the type-specific properties.
- For type Webhook, set these properties:
Webhook Property Description URL REST endpoint of the messaging platform to receive the alert notification. You can follow the setup steps in Slack Integration, VictorOps Integration, or HipChat Integration to obtain a notification URL. The notification URL must be publicly accessible. Content Type Content type of the POST body:
Custom Headers Name and value of one or more HTTP headers to pass in the POST request. Body Template Template describing the contents of the alert notification. Click Template and select the template that corresponds your messaging platform: Slack, VictorOps, or HipChat. Or, select Generic Webhook to see all of the available content options combined in a single template.
- For type Email, set these properties:
Email Property Description HTML Format Specifies whether the email platform should interpret the message body as HTML or plain text. When checked (the default), messages are interpreted as HTML. It is your responsibility to coordinate this setting with the chosen Body Template option. Email Address List One or more valid email addresses, separated by commas. Email Subject Subject of all emails from this alert target. Body Template Template describing the contents of the alert notification. Click Template and select the template that corresponds to your email formatting preference: HTML Email or Plain Text. It is your responsibility to coordinate this option with the HTML Format setting.
- For type PagerDuty, set these properties:
PagerDuty Property Description PagerDuty Key API integration key for the PagerDuty application. Follow the setup steps in PagerDuty Integration to obtain the key. Body Template Template describing the contents of the alert notification’s subject line. Click Template and select the PagerDuty Subject template.
- For type Webhook, set these properties:
- If you want to send notifications to different targets for different point tags, you can specify them under Recipients.
- The Default Recipients field specifies recipients that get all alerts.
The Routing field allows you to specify the following key/value pairs:
key value source <source name> metric <metric name> <point tag name> <point tag value>
Note: You must specify either default recipients or recipients determined by routing.
The screenshot below shows this for alert targets of type email.
- Optionally customize the Body Template using the variables and functions described in Customizing Alert Target Templates.
- Click Save to add the alert target and make it visible on the Alert Targets page.
- Test your new alert target, and then add it to an alert.
Test a Custom Alert Target
Test your alert target to ensure that it works properly.
- Select Browse > Alert Targets and find the target on the Alert Targets page.
- Click the three dots to the left of the alert target and select Test.
Add a Custom Alert Target to a Wavefront Alert
To add a custom alert target to a new or existing alert:
- Display the Create Alert page or the Edit Alert page.
- Scroll down to the Target List section.
- In the Alert Target field, start typing. A dropdown list appears that contains all available Wavefront alert targets that can be integrated to your alert.
- Select the alert target that you want to add, and click Save.
Edit a Custom Alert Target
You can change a custom alert target at any time.
To edit a alert, click the alert target name in the Alert Targets browser or click the three dots to the left of the alert target and select Edit.
Delete Custom Alert Targets
You can delete one or more custom alert targets by checking the checkboxes next to the alert targets and clicking the Trash icon at the top of the Alert Targets page. The trash icon is grayed out if you don’t have permission to delete any of the selected alert targets.
To delete one alert target, use the trash icon or click the three dots to the left of the alert target and select Delete.
Find an Alert Target ID
Each custom alert target has a unique ID that the system generates when you first create the alert target. To find the ID:
- Click Browse > Alert Targets.
In the Name column, note the ID of the alert target under the description.
Add Custom Alert Routes
By default, an alert notification is sent to all recipients that are specified in the alert target. It’s possible to customize an alert target, for example:
- If source=host1, send an email to userA and userB.
- If source=host2, send an email to userX
You can customize for each target type, and can route the notification based on
source, or a point tag key. See Step 6 in Create an Alert Target above.
Content of Routed Notifications
Routed notifications differ from non-routed notifications:
If you don’t use routing, all recipients receive a notification that starts when the triggering condition was met.
If you use alert routing, each recipient receives a notification when the triggering condition for that route was met, but the notification points to the whole alert.
Consider this example:
- An alert monitors CPU in a dev environment and a prod environment.
- You specify two alert targets:
- Target 1 sends a notification to email@example.com when the alert is triggered, that is, when CPU on either dev or prod goes above the threshold.
- Target 2 sends a notification to firstname.lastname@example.org when
env=dev, that is, when CPU goes above the threshold on any hosts in the dev environment.
- When the prod environment goes above the threshold, email@example.com receives a notification.
- When the dev environment goes above the threshold, firstname.lastname@example.org receives a notification, and the notification window starts when the alert fired, that is, when the CPU went above the threshold for prod.
It’s easy to find the information for dev in the chart associated with the notification because there’s a second event that shows when the alert was updated.
Query Responses of Webhook Alert Targets
Wavefront exposes response codes from webooks alert target calls as metrics:
~alert.webhooks.<webhook_id>.1xx ~alert.webhooks.<webhook_id>.2xx ~alert.webhooks.<webhook_id>.3xx ~alert.webhooks.<webhook_id>.4xx ~alert.webhooks.<webhook_id>.5xx
Note Wavefront does not expose response codes from the simpler alert targets (Email and PagerDuty).
The response codes indicate if a webhook call was successful and if the webhook generated a notification. You can query these metrics to determine if any webhooks are generating a problem response code. The metrics have the point tag
name = <webhook_name> so you can determine all the response codes for a particular webhook alert target:
If the response code of the webhook is anything other than 2xx, Wavefront creates an event with the name