Learn how to customize alert target templates.

You can use Wavefront alert targets to integrate alerts with many types of notification systems. Using Alert Targets to Send Alerts to Notification Systems explains how to create and manage alert targets.

Alert target customization variables and customization functions support customizing alert target templates. Wavefront alert target templates support Mustache syntax.

Alert Target Customization Iterators and Properties

You can customize the alert target using iterators and properties.

The iterator categories are:

  • failing
  • inMaintenance
  • newlyFailing
  • recovered

The iterators return three types of objects:

  • host - Affected source (host). The only value returned by XXXHosts iterators such as failingHosts or newlyFailingHosts.
  • series - Returned by XXXSeries iterators such as failingSeries or newlyFailingSeries.
    • host - Affected source (host).
    • label - Metric or aggregation.
    • tags - Point tags on the series.
  • alert series - Returned by XXXAlertSeries iterators such as failingAlertSeries or newlyFailingAlertSeries. You can further customize objects of type alert series, but not the other object types.
    • host - Affected source (host).
    • label - Metric or aggregation.
    • tags - Point tags on the series.
    • observed - Number of points returned by the alert condition.
    • firing - Number of points that satisfy the alert condition.
    • stats - Series statistics: first, last, min, max, and mean. These are values for the Display Expression associated with the alert. If you do not set the Display Expression, the iterator returns only the value that is associated with the alert condition. Because the condition that triggers the alert is always either 0 or not 0, that information is usually not useful.

Only the XXXAlertSeries and XXXfailingSeries iterators iterate through an empty source (host).

Alert targets support the following customization variables:

VariableDefinition
additionalInformation Text in the Additional Information section of the alert.
alertId ID of the alert that triggered the alert target.
alertTags List of tags associated with the alert that triggered the alert target.
condition Alert condition query.
createdTime Time the alert was created.
endedTime Time the alert ended (resolved).
errorMessage Message that is returned if condition query processing results in an error. This usually occurs when the alert is in an invalid state.
failingAlertSeries Iterator for alert series that are failing.
failingHosts Iterator for sources that are failing.
failingSeries Iterator for series that are failing.
hostsFailingMessage List of sources (hosts) that are failing, displayed as a message.
inMaintenanceAlertSeries Iterator for alert series whose sources are in a maintenance window.
inMaintenanceSeries Iterator for series whose sources are in a maintenance window.
inMaintenanceHosts Iterator for sources that are in a maintenance window.
name Name of the alert.
newlyFailingAlertSeries Iterator for alert series that are newly affected and added to the failingAlertSeries list.
newlyFailingSeries Iterator for series that are newly affected and added to the failingSeries list.
newlyFailingHosts Iterator for sources that are newly affected and added to the failingHosts list.
notificationId Unique ID of each notification sent to the alert target.
reason Trigger that caused the alert target to send the notification, e.g. Alert Opened or Alert Snoozed.
recoveredAlertSeries Iterator for alert series identifiers that recovered from the alert.
recoveredSeries Iterator for series that recovered from the alert.
recoveredHosts Iterator for sources that recovered from the alert.
severity Alert severity (e.g. INFO, SMOKE, WARN, SEVERE).
severityInfo A flag set to True if alert severity is set to INFO.
severitySmoke A flag set to True if alert severity is set to SMOKE.
severitySevere A flag set to True if alert severity is set to SEVERE.
severityWarn A flag set to True if alert severity is set to WARN.
sinceTime Time elapsed since the alert started firing.
snoozedUntilTime Time when a snoozed alert is scheduled to be unsnoozed.
startedTime Time the alert started firing.
subject Subject of the payload (usually for email). By default the subject concatenates alert severity, alert trigger, and alert name.
url Link to a chart that shows alert firing events or resolved events along with the alert condition.

Webhook Template Example

Here is a sample webhook alert target template:

{
  "alertId": "{{{alertId}}}",
  "notificationId": "{{{notificationId}}}",
  "reason": "{{{reason}}}",
  "name": "{{#jsonEscape}}{{{name}}}{{/jsonEscape}}",
  "severity": "{{{severity}}}",
  "severitySmoke": {{severitySmoke}},
  "severityInfo": {{severityInfo}},
  "severityWarn": {{severityWarn}},
  "severitySevere": {{severitySevere}},
  "condition": "{{#jsonEscape}}{{{condition}}}{{/jsonEscape}}",
  "url": "{{{url}}}",
  "createdTime": "{{{createdTime}}}",
  "startedTime": "{{{startedTime}}}",
  "sinceTime": "{{{sinceTime}}}",
  "endedTime": "{{{endedTime}}}",
  "snoozedUntilTime": "{{{snoozedUntilTime}}}",
  "subject": "{{#jsonEscape}}{{{subject}}}{{/jsonEscape}}",
  "hostsFailingMessage": "{{#jsonEscape}}{{{hostsFailingMessage}}}{{/jsonEscape}}",
  "errorMessage": "{{#jsonEscape}}{{{errorMessage}}}{{/jsonEscape}}",
  "additionalInformation": "{{#jsonEscape}}{{{additionalInformation}}}{{/jsonEscape}}",
  "failingSources": [
    {{#trimTrailingComma}}
      {{#failingHosts}}
        "{{{.}}}",
      {{/failingHosts}}
    {{/trimTrailingComma}}
  ],
  "inMaintenanceSources": [
    {{#trimTrailingComma}}
      {{#inMaintenanceHosts}}
        "{{{.}}}",
      {{/inMaintenanceHosts}}
    {{/trimTrailingComma}}
  ],
  "newlyFailingSources": [
    {{#trimTrailingComma}}
      {{#newlyFailingHosts}}
        "{{{.}}}",
      {{/newlyFailingHosts}}
    {{/trimTrailingComma}}
  ],
  "recoveredSources": [
    {{#trimTrailingComma}}
      {{#recoveredHosts}}
        "{{{.}}}",
      {{/recoveredHosts}}
    {{/trimTrailingComma}}
  ],
  "failingSeries": [
    {{#trimTrailingComma}}
      {{#failingSeries}}
        {{{.}}},
      {{/failingSeries}}
    {{/trimTrailingComma}}
  ],
  "inMaintenanceSeries": [
    {{#trimTrailingComma}}
      {{#inMaintenanceSeries}}
        {{{.}}},
      {{/inMaintenanceSeries}}
    {{/trimTrailingComma}}
  ],
  "newlyFailingSeries": [
    {{#trimTrailingComma}}
      {{#newlyFailingSeries}}
        {{{.}}},
      {{/newlyFailingSeries}}
    {{/trimTrailingComma}}
  ],
  "recoveredSeries": [
    {{#trimTrailingComma}}
      {{#recoveredSeries}}
        {{{.}}},
      {{/recoveredSeries}}
    {{/trimTrailingComma}}
  ]
}

Here is a sample alert target output generated with the template:

{
  "alertId": "1460761882996",
  "notificationId": "66dc2064-6bc1-437e-abe0-7c41afcd4aab",
  "reason": "ALERT_OPENED",
  "name": "Alert on Data rate (Test)",
  "severity": "SMOKE",
  "severitySmoke": true,
  "severityInfo": false,
  "severityWarn": false,
  "severitySevere": false,
  "condition": "rate(ts(~agent.points.2878.received)) > 4",
  "url": "https://yourcompany.wavefront.com/u/LPc1zR8k9X",
  "createdTime": "04/15/2016 23:11:22 0000",
  "startedTime": "09/12/2016 21:47:39 0000",
  "sinceTime": "09/12/2016 21:45:39 0000",
  "endedTime": "",
  "snoozedUntilTime": "",
  "subject": "[SMOKE] OPENED: Alert on Data rate ( Test)",
  "hostsFailingMessage": "localhost (~agent.points.2878.received)",
  "errorMessage": "",
  "additionalInformation": "An alert to test a webhook integration with HipChat",
  "failingSources": ["localhost"],
  "inMaintenanceSources": [],
  "newlyFailingSources": ["localhost"],
  "recoveredSources": [],
  "failingSeries": [["localhost", "~agent.points.2878.received", []]],
  "inMaintenanceSeries": [],
  "newlyFailingSeries": [["localhost", "~agent.points.2878.received", []]],
  "recoveredSeries": []
  }

Example: Accessing Series Values

Because an alert has a time window, a series does not have a single value when the threshold is crossed. For example, the alert might be set up to fire when a condition is true for 10 minutes. During a 10 minute period where the condition is true, the series likely have multiple values.

Sometimes you want to know the value of a series when an alert is triggered. For example, if the alert threshold is 80, you might want to know if the value was 81 or 91 on crossing the threshold. You might also want access to the value in the alert notification.

You can use the failingAlertSeries iterator to access series statistics—first, last, min, max, and mean—of the series values. The last statistic is automatically appended to email and PagerDuty messages.

The following template illustrates how to use the failingAlertSeries iterator to retrieve series statistics:

"failingAlertSeries": [
  {{#trimTrailingComma}}
    {{#failingAlertSeries}}
      "Source: {{host}}, Label: {{label}}, Tags: {{tags}}, Observed: {{observed}}, Firing: {{firing}},
      First: {{stats.first}}, Last: {{stats.last}}, Min: {{stats.min}}, Max: {{stats.max}}, Mean: {{stats.mean}}",
    {{/failingAlertSeries}}
  {{/trimTrailingComma}}
]

This template might yield the following message:

"failingAlertSeries": [
{"Source": "raspberrypi", "Label": "humidity", "Tags": {}, "Observed": 5, "Firing": 2, "First": 46.6, "Last": 46.0, "Min": 46.0, "Max": 46.6, "Mean": 46.279999999999994}}]

Alert Target Customization Functions

Customization functions let you set limits on the number of items returned by iterators. The service you use for notification might have a limit on the number of characters, and using customization functions helps you not exceed the limit.

The default value for each limit you can set with a customization function is 500. You must set the limit before iteration or the limit does not take effect.

The order of the limit settings determines limit precedence. For example, if you first set setDefaultIterationLimit and then you set setFailingLimit, then setFailingLimit overwrites the setDefaultIterationLimit setting.

The failingLimit property applies to all iterators in the failing category: failingAlertSeries, failingSeries, and failingHosts.

See Setting and Testing Iteration Limits for an example.

FunctionDefinition
getIterationLimit Gets the value of an iteration limit. Valid values are: defaultIterationLimit, failingLimit, inMaintenanceLimit, newlyFailingLimit, and recoveredLimit.
iterationLimitExceed Checks whether the number of the result returned is limited by an iteration limit. Valid values are: failingLimitExceed, inMaintenanceLimitExceed, newlyFailingLimitExceed, and recoveredLimitExceed.
setDefaultIterationLimit Sets all limits to the same value.
setFailingLimit Sets the limit for the number of items returned by failingAlertSeries, failingHosts, and failingSeries.
setInMaintenanceLimit Sets the limit for the number of items returned by inMaintenanceAlertSeries, inMaintenanceHosts, and inMaintenanceSeries.
setNewlyFailingLimit Sets the limit for the number of items returned by newlyFailingAlertSeries, newlyFailingHosts, and newlyFailingSeries.
setRecoveredLimit Sets the limit for the number of items returned by recoveredAlertSeries, recoveredHosts, and recoveredSeries.

Example: Setting and Testing Iteration Limits

Suppose you have 8 failing sources: “source1”, “source2”, “source3”, “source4”, “source5”, “source6”, “source7”, “source8”. You set setDefaultIterationLimit to 5 in the first line of the following template:

{{#setDefaultIterationLimit}}5{{/setDefaultIterationLimit}}
{
  "getIterationLimit": {
     "defaultIterationLimit": "{{{defaultIterationLimit}}}",
     "failingLimit": "{{{failingLimit}}}",
     "inMaintenanceLimit": "{{{inMaintenanceLimit}}}",
     "newlyFailingLimit": "{{{newlyFailingLimit}}}",
     "recoveredLimit": "{{{recoveredLimit}}}"
   },
   "iterationLimitExceed": {
     "failingLimitExceed": "{{{failingLimitExceed}}}",
     "inMaintenanceLimitExceed": "{{{inMaintenanceLimitExceed}}}",
     "newlyFailingLimitExceed": ""{{{newlyFailingLimitExceed}}}",
     "recoveredLimitExceed": "{{{recoveredLimitExceed}}}"
   },
  "alertId": "{{{alertId}}}",
  "alertTags": "[
    {{#trimTrailingComma}}
      {{#alertTags}}
        "{{#jsonEscape}}{{{.}}}{{/jsonEscape}}",
      {{/alertTags}}
    {{/trimTrailingComma}}
  ],
  ...
  "failingSources": [
    {{#trimTrailingComma}}
      {{#failingHosts}}
        "{{{.}}}",
      {{/failingHosts}}
    {{/trimTrailingComma}}
  ],
  "failingSeries": [
    {{#trimTrailingComma}}
      {{#failingSeries}}
        {{{.}}},
      {{/failingSeries}}
    {{/trimTrailingComma}}
  ]
}

The template with these settings results in the following payload for the 8 failing sources:

{
 "getIterationLimit": {
   "defaultIterationLimit": "5",
   "failingLimit": "5",
   "inMaintenanceLimit": "5",
   "newlyFailingLimit": "5",
   "recoveredLimit": "5"
 },
 "iterationLimitExceed": {
   "failingLimitExceed": "true",
   "inMaintenanceLimitExceed": "false",
   "newlyFailingLimitExceed": "false",
   "recoveredLimitExceed": "false"
 },
 "alertId": "1492543979795",
 "alertTags": [production, mysql],
 ...
 "failingSources": ["source5", "source4", "source7", "source6", "source1"],
 "failingSeries": [[null,"3.0",[]]]
}

failingHosts iterates only up to failingLimit, which is 5 in this case. failingLimitExceed is true because the number of failing sources exceeds the limit.

In contrast, if the failingLimit is 10, the payload is the following for 8 failing sources:

{
  "getIterationLimit": {
    "defaultIterationLimit": "10",
    "failingLimit": "10",
    "inMaintenanceLimit": "10",
    "newlyFailingLimit": "10",
    "recoveredLimit": "10"
  },
  "iterationLimitExceed": {
    "failingLimitExceed": "false",
    "inMaintenanceLimitExceed": "false",
    "newlyFailingLimitExceed": "false",
    "recoveredLimitExceed": "false"
  },
  "alertId": "1492543979795",
  "alertTags": [production, mysql],
  ...
  "failingSources": ["source5", "source4", "source7", "source6", "source1", "source3", "source2", "source8"],
  "failingSeries": [[null,"3.0",[]]]
}

For this case (limit 10, failing sources 8) failingLimitExceed is false because the number the failing sources does not exceed the limit set.

Alert Target Utility Functions

Alert target utility functions allow you to make the output of the alert target more readable. If you send the notification to a system that uses JSON, you can use jsonEscape. If the system uses XML, you can use xml11Escape or xml10Escape.

FunctionDefinitionExample
jsonEscape Escapes the characters in a string using Json string rules. Escapes any values it finds into their Json string form. Deals correctly with quotes and control-chars (tab, backslash, cr, ff, etc) so, for example, a tab becomes the characters \\ and t. Input: He didn't say, "Stop!"
Output: He didn't say, \"Stop!\"
xml11Escape Escapes the characters in a String using XML entities.
XML 1.1 can represent certain control characters, but it cannot represent the null byte or unpaired Unicode surrogate codepoints, even after escaping. escapeXml11 removes characters that do not fit in the following ranges:
[#x1-#xD7FF]|[#xE000-#xFFFD]|[#x10000-#x10FFFF]
`escapeXml11` escapes characters in the following ranges:
[#x1-#x8]|[#xB-#xC]|[#xE-#x1F]|[#x7F-#x84]|[#x86-#x9F]
Input: "bread" & "chocolate"
Output: "bread" & "chocolate"
xml10Escape
Escapes the characters in a string using XML entities. XML 1.0 is a text-only format, it cannot represent control characters or unpaired Unicode surrogate codepoints, even after escaping.
escapeXml10 removes characters that do not fit in the following ranges: #x9|#xA|#xD|[#x20-#xD7FF]|[#xE000-#xFFFD]|[#x10000-#x10FFFF]
escapeXml10 escapes characters in the following ranges: [#x7F-#x84]|[#x86-#x9F]
Input: "bread" & "chocolate"
Output: "bread" & "chocolate"
trimTrailingComma Retain the string content up to the last comma. Often used with webhook templates with iterators to remove the extra comma of the last item. Input:
"(Host: , Label: 3.0, Tags: , Observed: 5 Firing: 5, First: , Last: , Min: , Max: , Mean: ), "
Output:
"(Host: , Label: 3.0, Tags: , Observed: 5 Firing: 5, First: , Last: , Min: , Max: , Mean: )"