Learn how to use webhooks to integrate alerts and notification systems.

Wavefront uses webhooks to integrate alerts with many types of notification systems. A webhook is a user-defined HTTP callback triggered when an alert changes state. When the state change occurs, Wavefront makes an HTTP POST request to the URL configured for the webhook that contains data passed as a JSON payload.

This section describes how to create and configure webhooks and webhook payloads. You can use webhook templates, variables, and functions to construct virtually any type of payload.

To integrate alerts with Slack, PagerDuty, HipChat, or VictorOps, follow the instructions in the in-product integrations. Wavefront provides webhook templates for each of these systems.

To view and manage webhooks, select Browse > Webhooks.

Creating a Webhook

To create a webhook:

  1. Select Browse > Webhooks.
  2. Click the Create Webhook button.
  3. Fill in the webhook properties.
    PropertyDescription
    Name The name of the webhook. The name should be simple while still making it easy to identify its purpose.
    Triggers A set of one or more alert state changes that trigger the webhook. The options are:
    • Alert Opened - When the alert is opened (fired).
    • Alert Status Updated - When the status of an open alert changes. For example, a new source satisfies the alert condition and joins the set of affected sources.
    • Alert Resolved - When the alert is resolved.
    • Alert Affected by Maintenance Window - When the alert is affected by a maintenance window.
    • Alert Snoozed - When the alert is snoozed.
    • Alert Has No Data - When the series referenced in the alert condition is not reporting data.
    • Alert Has No Data Resolved - When the series referenced in the alert condition has started reporting data after having no data.
    • Alert Entered Maintenance From No Data - When the series referenced in the alert condition is not reporting data and is affected by a maintenance window.
    URL The REST endpoint of the receiving application, e.g. Slack.
    Content Type The content type of the POST body:
    • application/json
    • text/html
    • text/plain
    • application/x-www-form-urlencoded
    Custom Headers The name and value of one or more HTTP header to pass in the POST request.
    Webhook POST Body Template Template for a payload that the webhook sends in the POST request.
    Description The purpose of the webhook.
  4. Select Webhook POST Body Template > Template > <template_type>, where <template_type> is Default, Slack, VictorOps, and HipChat.
  5. Customize the template as described in the next section.
  6. Click Save. The webhook is added to the Webhooks page.

Customizing Webhook Templates

Wavefront webhook templates support Mustache syntax and a set of payload variables and functions.

Payload Variables

You can customize the payload using properties and iterators that characterize the alert triggering the webhook.

The categories of iterators are: failing, inMaintenance, newlyFailing, and recovered. The iterators return three types of objects:

  • host - the affected source (host). Returned by XXXHosts iterators.
  • series - Returned by XXXSeries iterators.
    • host - the affected source (host).
    • label - the metric or aggregation.
    • tags - the point tags on the series.
  • alert series - Returned by XXXAlertSeries iterators.
    • host - the affected source (host).
    • label - the metric or aggregation.
    • tags - the point tags on the series.
    • observed - the number of points returned by the alert condition.
    • firing - the number of points that satisfy the alert condition.
    • stats - series statistics: first, last, min, max, and mean. These are values for the Display Expression associated with the alert. If Display Expression is not set, stats returns the value associated with the condition, which is sometimes not meaningful. For example: ts(something.metric) > 30. The stats values will be either 1 or 0.

Only the failingAlertSeries and failingSeries iterators iterate through an empty source (host).

VariableDefinition
alertId The ID of the alert that triggered the webhook.
alertTags A list of tags associated with the alert that triggered the webhook.
condition The alert condition query.
createdTime Time the alert was created.
endedTime Time the alert ended(resolved).
errorMessage The message if there is an error while processing the condition query. This usually occurs when the alert is in an invalid state.
failingAlertSeries An iterator for alert series that are failing.
failingHosts An iterator for sources that are failing.
failingSeries An iterator for series that are failing.
hostFailingMessage The list of sources (hosts) that are failing displayed as a message.
inMaintenanceAlertSeries An iterator for alert series whose sources are in a maintenance window.
inMaintenanceSeries An iterator for series whose sources are in a maintenance window.
inMaintenanceHosts An iterator for sources that are in a maintenance window.
name The name of the alert.
newlyFailingAlertSeries An iterator for alert series that are newly affected and added to the failingAlertSeries list.
newlyFailingSeries An iterator for series that are newly affected and added to the failingSeries list.
newlyFailingHosts An iterator for sources that are newly affected and added to the failingHosts list.
notificationId A unique ID of each notification sent to the webhook.
reason Trigger that caused the webhook to send the notification: e.g. Alert Opened or Alert Snoozed.
recoveredAlertSeries An iterator for alert series identifiers that recovered from the alert.
recoveredSeries An iterator for series that recovered from the alert.
recoveredHosts An iterator for sources that recovered from the alert.
severity The alert severity (e.g. INFO, SMOKE, WARN, SEVERE).
severityInfo A flag set to True if the alert severity is set to INFO.
severitySmoke A flag set to True if the alert severity is set to SMOKE.
severitySevere A flag set to True if the alert severity is set to SEVERE.
severityWarn A flag set to True if the alert severity is set to WARN.
sinceTime Time elapsed since the alert started firing.
snoozedUntilTime Time that a snoozed alert is scheduled to be unsnoozed.
startedTime Time the alert started firing.
subject The subject of the payload (usually for email). By default it concatenates the alert severity, alert trigger, and alert name.
url A link to a chart that shows the alert firing or resolved events along with the alert condition.

Example

Here is a sample template:

{
  "alertId": "{{{alertId}}}",  
  "notificationId": "{{{notificationId}}}",  
  "reason": "{{{reason}}}",  
  "name": "{{#jsonEscape}}{{{name}}}{{/jsonEscape}}",  
  "severity": "{{{severity}}}",  
  "severitySmoke": {{severitySmoke}},  
  "severityInfo": {{severityInfo}},  
  "severityWarn": {{severityWarn}},  
  "severitySevere": {{severitySevere}},  
  "condition": "{{#jsonEscape}}{{{condition}}}{{/jsonEscape}}",  
  "url": "{{{url}}}",  
  "createdTime": "{{{createdTime}}}",  
  "startedTime": "{{{startedTime}}}",  
  "sinceTime": "{{{sinceTime}}}",  
  "endedTime": "{{{endedTime}}}",  
  "snoozedUntilTime": "{{{snoozedUntilTime}}}",  
  "subject": "{{#jsonEscape}}{{{subject}}}{{/jsonEscape}}",  
  "sourcesFailingMessage": "{{#jsonEscape}}{{{sourcesFailingMessage}}}{{/jsonEscape}}",  
  "errorMessage": "{{#jsonEscape}}{{{errorMessage}}}{{/jsonEscape}}",  
  "additionalInformation": "{{#jsonEscape}}{{{additionalInformation}}}{{/jsonEscape}}",  
  "failingSources": [  
    {{#trimTrailingComma}}  
      {{#failingHosts}}  
        "{{{.}}}",  
      {{/failingHosts}}  
    {{/trimTrailingComma}}  
  ],  
  "inMaintenanceSources": [  
    {{#trimTrailingComma}}  
      {{#inMaintenanceHosts}}  
        "{{{.}}}",  
      {{/inMaintenanceHosts}}  
    {{/trimTrailingComma}}  
  ],  
  "newlyFailingSources": [  
    {{#trimTrailingComma}}  
      {{#newlyFailingHosts}}  
        "{{{.}}}",  
      {{/newlyFailingHosts}}  
    {{/trimTrailingComma}}  
  ],  
  "recoveredSources": [  
    {{#trimTrailingComma}}  
      {{#recoveredHosts}}  
        "{{{.}}}",  
      {{/recoveredHosts}}  
    {{/trimTrailingComma}}  
  ],  
  "failingSeries": [  
    {{#trimTrailingComma}}  
      {{#failingSeries}}  
        {{{.}}},  
      {{/failingSeries}}  
    {{/trimTrailingComma}}  
  ],  
  "inMaintenanceSeries": [  
    {{#trimTrailingComma}}  
      {{#inMaintenanceSeries}}  
        {{{.}}},  
      {{/inMaintenanceSeries}}  
    {{/trimTrailingComma}}  
  ],  
  "newlyFailingSeries": [  
    {{#trimTrailingComma}}  
      {{#newlyFailingSeries}}  
        {{{.}}},  
      {{/newlyFailingSeries}}  
    {{/trimTrailingComma}}  
  ],  
  "recoveredSeries": [  
    {{#trimTrailingComma}}  
      {{#recoveredSeries}}  
        {{{.}}},  
      {{/recoveredSeries}}  
    {{/trimTrailingComma}}  
  ]  
}

Here is a sample payload for the template:

{
  "alertId": "1460761882996",
  "notificationId": "66dc2064-6bc1-437e-abe0-7c41afcd4aab",
  "reason": "ALERT_OPENED",
  "name": "Alert on Data rate (Test)",
  "severity": "SMOKE",
  "severitySmoke": true,
  "severityInfo": false,
  "severityWarn": false,
  "severitySevere": false,
  "condition": "rate(ts(~agent.points.2878.received)) > 4",
  "url": "https://yourcompany.wavefront.com/u/LPc1zR8k9X",
  "createdTime": "04/15/2016 23:11:22 0000",
  "startedTime": "09/12/2016 21:47:39 0000",
  "sinceTime": "09/12/2016 21:45:39 0000",
  "endedTime": "",
  "snoozedUntilTime": "",
  "subject": "[SMOKE] OPENED: Alert on Data rate ( Test)",
  "sourcesFailingMessage": "localhost (~agent.points.2878.received)",
  "errorMessage": "",
  "additionalInformation": "An alert to test a webhook integration with HipChat",
  "failingSources": ["localhost"],
  "inMaintenanceSources": [],
  "newlyFailingSources": ["localhost"],
  "recoveredSources": [],
  "failingSeries": [["localhost", "~agent.points.2878.received", []]],
  "inMaintenanceSeries": [],
  "newlyFailingSeries": [["localhost", "~agent.points.2878.received", []]],
  "recoveredSeries": []
  }

Example: Accessing Series Values

Sometimes you want to know the value of a series when an alert is triggered. For example, if the alert threshold is 80, you want to know on crossing the threshold if the value was 81 or 91. Furthermore you would like access to the value in the alert notification. Since an alert has a time window, a series does not have a single value when the threshold is crossed. For example, the alert may specify that the alert should fire when a condition is true for 10 minutes. During that 10 minute period, the series will likely have multiple values.

You can use the failingAlertSeries iterator to access series statistics—first, last, min, max, and mean—of the series values. The last statistic is automatically appended to email and PagerDuty messages.

The following template illustrates how to use the failingAlertSeries iterator to retrieve series statistics:

"failingAlertSeries": [
  {{#trimTrailingComma}}
    {{#failingAlertSeries}}
      "Source: {{host}}, Label: {{label}}, Tags: {{tags}}, Observed: {{observed}}, Firing: {{firing}},
      First: {{stats.first}}, Last: {{stats.last}}, Min: {{stats.min}}, Max: {{stats.max}}, Mean: {{stats.mean}}",
    {{/failingAlertSeries}}
  {{/trimTrailingComma}}
]

This template could yield the following message:

"failingAlertSeries": [
{"Source": "raspberrypi", "Label": "humidity", "Tags": {}, "Observed": 5, "Firing": 2, "First": 46.6, "Last": 46.0, "Min": 46.0, "Max": 46.6, "Mean": 46.279999999999994}}]

Payload Functions

Payload functions let you set limits on the number of items returned by iterators. The default value for each limit is 500. For a limit to take effect, the limit must be set before iteration.

The order of the limit settings determines limit precedence. For example, setting setDefaultIterationLimit after setting setFailingLimit overwrites the setFailingLimit setting.

The failingLimit property applies to all iterators in the failing category: failingAlertSeries, failingSeries, and failingHosts.

For a payload function example, see Setting and Testing Iteration Limits.

FunctionDefinition
setDefaultIterationLimit Sets all limits to the same value.
setFailingLimit Set the limit for the number of items returned by failingAlertSeries, failingHosts, and failingSeries.
setInMaintenanceLimit Set the limit for the number of items returned by inMaintenanceAlertSeries, inMaintenanceHosts, and inMaintenanceSeries.
setNewlyFailingLimit Set the limit for the number of items returned by newlyFailingAlertSeries, newlyFailingHosts, and newlyFailingSeries.
setRecoveredLimit Set the limit for the number of items returned by recoveredAlertSeries, recoveredHosts, and recoveredSeries.
getIterationLimit Gets the value of an iteration limit. Valid values are: defaultIterationLimit, failingLimit, inMaintenanceLimit, newlyFailingLimit, and recoveredLimit.
iterationLimitExceed Check whether an iteration limit is limiting the number of the result returned. Valid values are: failingLimitExceed, inMaintenanceLimitExceed, newlyFailingLimitExceed, and recoveredLimitExceed.

Example: Setting and Testing Iteration Limits

Suppose you have 8 failing sources: “source1”, “source2”, “source3”, “source4”, “source5”, “source6”, “source7”, “source8”. Setting setDefaultIterationLimit to 5 in the following template:

{{#setDefaultIterationLimit}}5{{/setDefaultIterationLimit}}
{
  "getIterationLimit": {
     "defaultIterationLimit": "{{{defaultIterationLimit}}}",
     "failingLimit": "{{{failingLimit}}}",
     "inMaintenanceLimit": "{{{inMaintenanceLimit}}}",
     "newlyFailingLimit": "{{{newlyFailingLimit}}}",
     "recoveredLimit": "{{{recoveredLimit}}}"
   },
   "iterationLimitExceed": {
     "failingLimitExceed": "{{{failingLimitExceed}}}",
     "inMaintenanceLimitExceed": "{{{inMaintenanceLimitExceed}}}",
     "newlyFailingLimitExceed": ""{{{newlyFailingLimitExceed}}}",
     "recoveredLimitExceed": "{{{recoveredLimitExceed}}}"
   },
  "alertId": "{{{alertId}}}",
  "alertTags": "[
    {{#trimTrailingComma}}
      {{#alertTags}}
        "{{#jsonEscape}}{{{.}}}{{/jsonEscape}}",
      {{/alertTags}}
    {{/trimTrailingComma}}
  ],
  ...
  "failingSources": [  
    {{#trimTrailingComma}}  
      {{#failingHosts}}  
        "{{{.}}}",  
      {{/failingHosts}}  
    {{/trimTrailingComma}}  
  ], 
  "failingSeries": [  
    {{#trimTrailingComma}}  
      {{#failingSeries}}  
        {{{.}}},  
      {{/failingSeries}}  
    {{/trimTrailingComma}}  
  ]
}

Results in the following payload:

{
 "getIterationLimit": {
   "defaultIterationLimit": "5",
   "failingLimit": "5",
   "inMaintenanceLimit": "5",
   "newlyFailingLimit": "5",
   "recoveredLimit": "5"
 },
 "iterationLimitExceed": {
   "failingLimitExceed": "true",
   "inMaintenanceLimitExceed": "false",
   "newlyFailingLimitExceed": "false",
   "recoveredLimitExceed": "false"
 },
 "alertId": "1492543979795",
 "alertTags": [production, mysql],
 ...
 "failingSources": ["source5", "source4", "source7", "source6", "source1"],
 "failingSeries": [[null,"3.0",[]]]
}

failingHosts iterates only up to failingLimit, which in this case is 5. failingLimitExceed is true because the number the failing sources exceeds the limit set. In the case in which the limit is 10, the payload is:

{ 
  "getIterationLimit": {
    "defaultIterationLimit": "10",
    "failingLimit": "10",
    "inMaintenanceLimit": "10",
    "newlyFailingLimit": "10",
    "recoveredLimit": "10"
  },
  "iterationLimitExceed": {
    "failingLimitExceed": "false",
    "inMaintenanceLimitExceed": "false",
    "newlyFailingLimitExceed": "false",
    "recoveredLimitExceed": "false"
  },
  "alertId": "1492543979795",
  "alertTags": [production, mysql],
  ...
  "failingSources": ["source5", "source4", "source7", "source6", "source1", "source3", "source2", "source8"],
  "failingSeries": [[null,"3.0",[]]]
}

failingLimitExceed is false because the number the failing sources does not exceed the limit set.

Testing a Webhook

To check that a webhook is functioning properly you may want to test it first. To test a webhook, select action_menu > Test at the far right of the webhook.

Querying Webhook Responses

In order to determine if a webhook call was successful and a notification was generated via the webhook, Wavefront exposes response codes from webhook calls as metrics:

alert.webhooks.<webhook_id>.1xx
alert.webhooks.<webhook_id>.2xx
alert.webhooks.<webhook_id>.3xx
alert.webhooks.<webhook_id>.4xx
alert.webhooks.<webhook_id>.5xx

By querying these metrics, you can determine which webhooks are generating a problem response code. The metrics have the point tag name = <webhook_name> so you can determine all the response codes for a particular webhook:

ts(alert.webhooks.*.*, name=<webhook_name>)

If the response code of the webhook is anything other than 2xx, an event with the name <webhook_id>.<webhook_name>.<response_code> is created.

Editing a Webhook

To edit a webhook, click the webhook name in the Webhooks browser or select action_menu > Edit at the far right of the webhook.

Deleting Webhooks

You can delete one or more webhooks by checking the checkboxes next to the webhooks and clicking the Trash icon at the top of the Webhooks page. The Trash icon is grayed out if any of the selected webhooks cannot be deleted.

To delete one webhook, select action menu > Delete at the far right of the webhook.

Finding a Webhook ID

Each webhook has a unique system generated ID. Such IDs are used when adding a webhook as an alert target. To find the ID:

  1. Click Browse > Webhooks.
  2. Search for the webhook. In the Name column, note the ID of the webhook under the webhook description.

    webhook ID

Adding a Webhook to a Wavefront Alert

To add a webhook as the target of an alert:

  1. Click Alerts or select Browse > Alerts.
  2. Locate the alert in the Alerts page and click the alert name.
  3. Scroll down to the Targets field.
  4. Add the keyword webhook to the targets list. A dropdown list displays containing all the available webhooks present in Wavefront that can be integrated to your alert. The ID and the webhook URL for each webhook is listed.
  5. Locate and select the ID of the webhook noted in Finding a Webhook ID.
  6. Click Save.