You can create custom alert targets to configure alert notifications for a variety of messaging platforms, including email, pager services, and communication channels.
This page provides general steps for creating and managing a custom alert target.
As a convenience, you can follow the steps in PagerDuty Integration, VictorOps Integration, Slack Integration, or HipChat Integration to configure custom alert targets specifically for these messaging platforms. (You can check for additional integrations here.)
See Customizing Alert Target Templates for specific details about customizing the contents of alert notifications.
Our blog post Engineering Tips Series: How Wavefront’s Devops Team Uses Alert Targets to Provide Exceptional Quality of Services to Customers explains how alert targets help Wavefront to keep things running smoothly.
About Alert Targets
Alert targets specify when and how to send notifications in response to alert activity. You implicitly use Wavefront’s built-in alert targets when you specify an email address or a PagerDuty key in the target list of an alert. These simple alert targets:
- Cause notifications to be sent whenever the alert is firing, updated, resolved, snoozed or in a maintenance window.
- Provide internal templates that compose and format the notification contents. These internal templates are maintained by Wavefront, and may change from release to release.
You can create custom alert targets when you need more flexibility about where you want notifications to be sent, what kind of information you want in them, how you want them to be formatted, and which alert events should trigger them. For example, you could use a custom alert target to:
- Expand (or limit) the set of triggering events for notifications.
- Configure different contents for notifications triggered by different events.
- Associate a short name with a long list of email addresses or a lengthy PagerDuty key.
Note: You always use a custom alert target for sending notifications to a webhook-based messaging platform such as Slack, HipChat, or VictorOps.
Viewing Custom Alert Targets
To view alert targets, select Browse > Alert Targets.
Creating a Custom Alert Target
The process for creating an alert target is similar for the different types of targets. Setting the Type causes a few different fields to be displayed.
- Select Browse > Alert Targets.
- Click the Create Alert Target button.
- Fill in the properties that are common to all alert targets.
Property Description Name Name of the alert target. Pick a name that is simple and that makes it easy to identify the alert target's purpose. Description Description of the alert target. Triggers One or more alert state changes that trigger the alert target. The options are:
- Alert Firing - Trigger when the alert transitions from checking to firing.
- Alert Status Updated - Trigger when at least one time series changes category while the alert continues firing. For example, an individual time series could start to fail (satisfy the alert condition during the Alert fires time window) or could recover (stop satisfying the alert condition during the Alert resolves time window).
- Alert Resolved - Trigger when the alert resolves.
- Alert Affected by Maintenance Window - Trigger when the alert is affected by a maintenance window.
- Alert Snoozed - Trigger when the alert is snoozed.
- Alert Has No Data - Trigger when the time series associated with the alert have all stopped reporting data.
- Alert Has No Data Resolved - Trigger when at least one time series associated with the alert has started reporting data, while all other time series are still reporting no data.
- Alert Entered Maintenance From No Data - Trigger when none of the alert's time series are reporting data, and the alert is affected by a maintenance window.
- From the Type pull-down menu, select the alert target type:
Type Description Webhook Alert target for sending notifications to messaging platforms such as Slack, VictorOps, or HipChat. This alert target defines the HTTP callback (POST request and URL) that is triggered when an alert changes state. Alert target for sending notifications to email systems. This alert target specifies the attributes of the email messages to be sent when an alert changes state. PagerDuty Alert target for sending notifications to PagerDuty. This alert target specifies the PagerDuty key and a POST body to use when an alert changes state.
- Fill in the type-specific properties.
- For type Webhook, set these properties:
Webhook Property Description URL REST endpoint of the messaging platform to receive the alert notification. You can follow the setup steps in Slack Integration, VictorOps Integration, or HipChat Integration to obtain a notification URL. The notification URL must be publicly accessible. Content Type Content type of the POST body:
Custom Headers Name and value of one or more HTTP headers to pass in the POST request. Body Template Template describing the contents of the alert notification. Click Template and select the template that corresponds your messaging platform: Slack, VictorOps, or HipChat. Or, select Generic Webhook to see all of the available content options combined in a single template.
- For type Email, set these properties:
Email Property Description HTML Format Specifies whether the email platform should interpret the message body as HTML or plain text. When checked (the default), messages are interpreted as HTML. It is your responsibility to coordinate this setting with the chosen Body Template option. Email Address List One or more valid email addresses, separated by commas. Email Subject Subject of all emails from this alert target. Body Template Template describing the contents of the alert notification. Click Template and select the template that corresponds to your email formatting preference: HTML Email or Plain Text. It is your responsibility to coordinate this option with the HTML Format setting.
- For type PagerDuty, set these properties:
PagerDuty Property Description PagerDuty Key API integration key for the PagerDuty application. Follow the setup steps in PagerDuty Integration to obtain the key. Body Template Template describing the contents of the alert notification’s subject line. Click Template and select the PagerDuty Subject template.
- For type Webhook, set these properties:
- Optionally customize the Body Template using the variables and functions described in Customizing Alert Target Templates.
- Click Save to add the alert target and make it visible on the Alert Targets page.
- Test your new alert target, and then add it to an alert.
Testing a Custom Alert Target
Test your alert target to ensure that it works properly.
- Select Browse > Alert Targets and find the target on the Alert Targets page.
- Click the three dots to the left of the alert target and select Test.
Adding a Custom Alert Target to a Wavefront Alert
To add a custom alert target to a new or existing alert:
- Display the Create Alert page or the Edit Alert page.
- Scroll down to the Target List section.
- In the Alert Target field, start typing. A dropdown list appears that contains all available Wavefront alert targets that can be integrated to your alert.
- Select the alert target that you want to add, and click Save.
Editing a Custom Alert Target
You can change a custom alert target at any time.
To edit a alert, click the alert target name in the Alert Targets browser or click the three dots to the left of the alert target and select Edit.
Deleting Custom Alert Targets
You can delete one or more custom alert targets by checking the checkboxes next to the alert targets and clicking the Trash icon at the top of the Alert Targets page. The trash icon is grayed out if you don’t have permission to delete any of the selected alert targets.
To delete one alert target, use the trash icon or click the three dots to the left of the alert target and select Delete.
Finding an Alert Target ID
Each custom alert target has a unique ID that the system generates when you first create the alert target. To find the ID:
- Click Browse > Alert Targets.
In the Name column, note the ID of the alert target under the description.
Querying Responses of Webhook Alert Targets
Wavefront exposes response codes from webooks alert target calls as metrics:
~alert.webhooks.<webhook_id>.1xx ~alert.webhooks.<webhook_id>.2xx ~alert.webhooks.<webhook_id>.3xx ~alert.webhooks.<webhook_id>.4xx ~alert.webhooks.<webhook_id>.5xx
Note Wavefront does not expose response codes from the simpler alert targets (Email and PagerDuty).
The response codes indicate if a webhook call was successful and if the webhook generated a notification. You can query these metrics to determine if any webhooks are generating a problem response code. The metrics have the point tag
name = <webhook_name> so you can determine all the response codes for a particular webhook alert target:
If the response code of the webhook is anything other than 2xx, Wavefront creates an event with the name