Learn how to customize alert target templates.

A custom alert target provides a template for specifying the contents of the notifications to be sent when an alert changes state. The template is a blueprint for extracting information from the alert and assembling the notification.

To customize a template, start with the default template for the alert target type, and edit that template.

Note For general information about setting up custom alert targets, see Creating and Managing Custom Alert Targets.

About Alert Target Templates

The template defined by a custom alert target describes the contents of the notifications that will be sent whenever an alert transition triggers an event of interest. A template identifies the information you want to extract from the transitioned alert, and embeds that information in a formatted structure that is sent to the target’s messaging platform. The messaging platform interprets the structure and displays it as a readable notification. For example:

  • A template for an HTML email alert target specifies the HTML structure that will be sent as the message body, and specifies the alert information to be included within that structure. Each piece of alert information is inserted as a value of an HTML element or attribute.

  • A template for a Slack alert target specifies the JSON structure that will be POSTed to the Slack endpoint, and specifies the alert information to be included within that structure. Each piece of alert information is inserted as a value of a Slack-defined JSON attribute.

A custom alert target’s template uses Mustache syntax to combine literal text with Wavefront-defined variables and functions to produce the structures to be sent to the receiving messaging platform.

Predefined Templates

Wavefront provides a predefined template for each type of custom alert target. You can use the predefined template as is, or you can customize it to add, remove, or rearrange alert information and structural elements. You can even use a predefined template as a guide for composing your own template.

You can inspect a predefined template to see:

  • The Wavefront-defined variables and functions that extract information from the alert.

  • The structural elements in which the extracted information is embedded. These are JSON attributes, HTML elements, or plain text, depending on the messaging platform to which notifications will be sent.

The predefined Slack, HipChat, and VictorOps templates contain JSON attributes defined by the messaging platform. See the product documentation for the platform for details.

Note The predefined Webhook Generic template contains JSON attributes that do not conform to any particular message platform’s Webhook endpoint specification. This template simply demonstrates how to access the different kinds of alert information for a Webhook endpoint.

Template Variables

Wavefront defines template variables for accessing information about the alert and about the time series tested by the alert. When the alert triggers a notification, Wavefront replaces the variables in the template with strings that represent the requested values. 

How you use a variable depends on whether it is:

  • A property, which accesses a single value. For example, alertID accesses a single string that represents the alert’s unique ID.
  • An iterator, which accesses a value that is a list of elements. For example, alertTags accesses a list of 0 or more strings that represents tags associated with the alert.

The following snippet shows the basic Mustache syntax for a property and an iterator:

{{{alertId}}}     {{! a property}}

{{#alertTags}}    {{! an iterator}}
    {{{.}}}
{{/alertTags}}

Mustache supports several variations in each case, but this example shows the most commonly used syntax in the alert target templates:

  • Each property is enclosed in 3 pairs of curly braces. (In an HTML email template, you can use 2 pairs of curly braces around a property.)

  • Each iterator is used in a Mustache section, with the iterator’s name appearing on either end. Because an iterator successively visits each element in its list, you can use {{{.}}} within the section to indicate the element currently being visited. The section normally contains additional literal text and functions to format each visited element, such as a following comma or other separator character. Examples are shown below.

Template Functions

Wavefront defines template functions for performing various tasks, such as tailoring the notification content to the trigger type, limiting the number of elements an iterator can return, and assisting with JSON or XML formatting.

The following snippet shows the basic Mustache syntax for two functions:

{{! a function}}
{{#setDefaultIterationLimit}}5{{/setDefaultIterationLimit}}

{{! another function}}
{{#isAlertOpened}}
    ... {{! lines describing the message content }}
{{/isAlertOpened}}

Like iterators, a function is used in a Mustache section, with the function’s name appearing on either end. The contents of the section are passed as input to the function.

Displaying and Editing Predefined Templates

To display and edit a predefined template for a new or existing custom alert target:

  1. Create or view the custom alert target.
  2. In the Body Template field, click Template and select one of the predefined templates.
  3. Add, remove, or rearrange alert information and structural elements of the predefined template, or replace the predefined template with a template of your own.
  4. Scroll to the end of the Body Template field and click Save.

The custom alert target’s Type determines the list of available predefined templates. For example, here are the choices for a custom alert target of type Email:

predefined alert target

Obtaining Information About the Alert

Wavefront defines variables for obtaining information about the alert as a whole, such as the alert ID, timing, severity, and so on. Each of these variables is a property unless explicitly described as an iterator.

VariableDefinition
additionalInformation Text in the Additional Information section of the alert.
alertId ID of the alert that triggered the alert target.
alertTags Iterator returning 0 or more tags associated with the alert that triggered the alert target.
condition Alert condition query.
createdTime Time the alert was created.
endedTime Time the alert ended (resolved).
errorMessage Message that is returned if condition query processing results in an error. This usually occurs when the alert is in an invalid state.
hostsFailingMessage Message containing a list of the sources of the failing time series. These are time series for which the alert condition returned all true (non-zero) values for the duration of the Alert fires time window.
imageLinks Iterator returning URLs to chart images. Currently returns 1 URL to the chart image that shows the alert’s display expression at the time the alert fired or was updated.
name Name of the alert.
notificationId Unique ID of the notification being sent to the alert target.
reason Trigger that caused the alert target to send the notification, e.g. Alert Opened or Alert Snoozed.
severity Alert severity (e.g. INFO, SMOKE, WARN, SEVERE).
severityInfo A flag set to True if alert severity is set to INFO.
severitySmoke A flag set to True if alert severity is set to SMOKE.
severitySevere A flag set to True if alert severity is set to SEVERE.
severityWarn A flag set to True if alert severity is set to WARN.
sinceTime Time elapsed since the alert started firing.
snoozedUntilTime Time when a snoozed alert is scheduled to be unsnoozed.
startedTime Time the alert started firing.
subject Subject of the notification (usually for email). If you omit this variable, the subject is composed of the alert severity, alert trigger, and alert name.
url Link to an interactive chart that shows alert firing events or resolved events along with the alert condition.

Example: Accessing Alert Information in a Generic Webhook Alert Target Template

This portion of the Generic Webhook alert target template uses variables that access information about the alert.

{
  "alertId": "{{{alertId}}}",
  "alertTags": [
    {{#trimTrailingComma}}
      {{#alertTags}}
        "{{#jsonEscape}}{{{.}}}{{/jsonEscape}}",
      {{/alertTags}}
    {{/trimTrailingComma}}
  ],
  "notificationId": "{{{notificationId}}}",
  "imageLinks": "{{{imageLinks}}}",
  "reason": "{{{reason}}}",
  "name": "{{#jsonEscape}}{{{name}}}{{/jsonEscape}}",
  "severity": "{{{severity}}}",
  "severitySmoke": {{severitySmoke}},
  "severityInfo": {{severityInfo}},
  "severityWarn": {{severityWarn}},
  "severitySevere": {{severitySevere}},
  "condition": "{{#jsonEscape}}{{{condition}}}{{/jsonEscape}}",
  "url": "{{{url}}}",
  "createdTime": "{{{createdTime}}}",
  "startedTime": "{{{startedTime}}}",
  "sinceTime": "{{{sinceTime}}}",
  "endedTime": "{{{endedTime}}}",
  "snoozedUntilTime": "{{{snoozedUntilTime}}}",
  "subject": "{{#jsonEscape}}{{{subject}}}{{/jsonEscape}}",
  "hostsFailingMessage": "{{#jsonEscape}}{{{hostsFailingMessage}}}{{/jsonEscape}}",
  "errorMessage": "{{#jsonEscape}}{{{errorMessage}}}{{/jsonEscape}}",
  "additionalInformation": "{{#jsonEscape}}{{{additionalInformation}}}{{/jsonEscape}}"
}

Example: Alert Information in Output From the Sample Template

Here is sample alert target output generated with the preceding template:

{
  "alertId": "1460761882996",
  "alertTags": ["production", "mysql"],
  "notificationId": "66dc2064-6bc1-437e-abe0-7c41afcd4aab",
  "imageLinks": "[https://yourcompany.wavefront.com/api/v2/image/RPx3zR7u2X]",
  "reason": "ALERT_OPENED",
  "name": "Alert on Data rate (Test)",
  "severity": "SMOKE",
  "severitySmoke": true,
  "severityInfo": false,
  "severityWarn": false,
  "severitySevere": false,
  "condition": "rate(ts(~agent.points.2878.received)) > 4",
  "url": "https://yourcompany.wavefront.com/u/LPc1zR8k9X",
  "createdTime": "04/15/2016 23:11:22 0000",
  "startedTime": "09/12/2016 21:47:39 0000",
  "sinceTime": "09/12/2016 21:45:39 0000",
  "endedTime": "",
  "snoozedUntilTime": "",
  "subject": "[SMOKE] OPENED: Alert on Data rate ( Test)",
  "hostsFailingMessage": "localhost (~agent.points.2878.received)",
  "errorMessage": "",
  "additionalInformation": "An alert to test a webhook integration with HipChat"
  }

Notice that, in a template entry such as "alertId": "{{{alertId}}}", everything except the variable is literal text that is passed through as output. So, for example:

  • "alertId": " " is literal text that produces a sample JSON attribute called "alertId".
  • {{{alertId}}} invokes the Wavefront-defined variable alertId, which expands to 1460761882996 in our example.

Obtaining Information About the Alert’s Time Series

Wavefront defines variables for obtaining information about the time series that contributed to the alert’s state change. Each of these variables is an iterator that visits the time series in a particular category, and returns one of the following kinds of information about the visited series:

The time series visited by a particular iterator are in one of the following categories:

Time Series CategoryDefinition
Failing The time series that caused the alert to fire or update. These are time series for which the alert condition returned all true (non-zero) values for the duration of the Alert fires time window.
Newly failing Any time series that failed after the alert started firing, causing the alert to be updated. These are time series for which the alert condition returned all true (non-zero) values for the duration of the Alert fires time window, while at least one other time series continues to fail.
Recovered Any previously failing time series that is no longer failing, causing the alert to be updated or possibly resolved. These are time series for which the alert condition returned all true (non-zero) values for the duration of the Alert fires time window, and then returned either false (0) values or no data for the duration of the Alert resolves time window.
In maintenance Any time series whose source is associated with an ongoing maintenance window. These are time series that continue to be tested against the alert condition, but whose results do not change the alert’s state.

Note The names of the iterators follow this convention: <seriesCategory><InfoIndicator>. For example:

  • failingHosts is an iterator that lists the host name of each failing time series.
  • inMaintenanceSeries is an iterator that lists the defining information of each time series whose source is in maintenance.
  • recoveredAlertSeries is an iterator that can access a custom combination of details about each recovered time series.

Listing the Sources of an Alert’s Time Series

You can use the following iterators to visit each time series in the indicated category and return the string name of the series’ source (host). Any time series not associated with a source is skipped.

IteratorDefinition
failingHosts Iterator that returns the source of each failing time series.
inMaintenanceHosts Iterator that returns each source that is in a maintenance window.
newlyFailingHosts Iterator that returns the source of each time series that has failed since the previous notification. (These source names are also returned by failingHosts.)
recoveredHosts Iterator that returns the source of each time series that has recovered since the previous notification.

Example: Accessing Alert Sources in a Generic Webhook Alert Target Template

This portion of the Generic Webhook alert target template shows iterators that return the sources of the time series tested by the alert:

{
  "failingSources": [
    {{#trimTrailingComma}}
      {{#failingHosts}}
        "{{{.}}}",
      {{/failingHosts}}
    {{/trimTrailingComma}}
  ],
  "inMaintenanceSources": [
    {{#trimTrailingComma}}
      {{#inMaintenanceHosts}}
        "{{{.}}}",
      {{/inMaintenanceHosts}}
    {{/trimTrailingComma}}
  ],
  "newlyFailingSources": [
    {{#trimTrailingComma}}
      {{#newlyFailingHosts}}
        "{{{.}}}",
      {{/newlyFailingHosts}}
    {{/trimTrailingComma}}
  ],
  "recoveredSources": [
    {{#trimTrailingComma}}
      {{#recoveredHosts}}
        "{{{.}}}",
      {{/recoveredHosts}}
    {{/trimTrailingComma}}
  ]
}

Example: Alert Sources in Output From the Sample Template

Here is a sample alert target output generated with the preceding template:

{
  "failingSources": ["localhost", "db-1"],
  "inMaintenanceSources": ["app-3"],
  "newlyFailingSources": ["localhost", "db-1"],
  "recoveredSources": []
  }

Notice that the template provides literal text for enclosing each source name in quotation marks, for separating the source names with commas, and for enclosing the list in square brackets. The trimTrailingComma function suppresses the comma after the last source name.

Listing the Definitions of an Alert’s Time Series

You can use the following iterators to visit each time series in the indicated category and return the series’ defining information. The defining information for a series is a preformatted string that contains the source name, the metric name, and any point tags (shown as <key>=<value> pairs).

IteratorDefinition
failingSeries Iterator that returns the source, metric name, and point tags of each failing time series.
inMaintenanceSeries Iterator that returns the source, metric name, and point tags of each time series whose source is in a maintenance window.
newlyFailingSeries Iterator that returns the source, metric name, and point tags of each time series that has failed since the previous notification. (These series are also visited by failingSeries.)
recoveredSeries Iterator that returns the source, metric name, and point tags of each time series that has recovered since the previous notification.

Example: Accessing Time Series Information in a Generic Webhook Alert Target Template

This portion of the Generic Webhook alert target template shows iterators that return the defining information about the time series tested by the alert:

{
  "failingSeries": [
    {{#trimTrailingComma}}
      {{#failingSeries}}
        {{{.}}},
      {{/failingSeries}}
    {{/trimTrailingComma}}
  ],
  "inMaintenanceSeries": [
    {{#trimTrailingComma}}
      {{#inMaintenanceSeries}}
        {{{.}}},
      {{/inMaintenanceSeries}}
    {{/trimTrailingComma}}
  ],
  "newlyFailingSeries": [
    {{#trimTrailingComma}}
      {{#newlyFailingSeries}}
        {{{.}}},
      {{/newlyFailingSeries}}
    {{/trimTrailingComma}}
  ],
  "recoveredSeries": [
    {{#trimTrailingComma}}
      {{#recoveredSeries}}
        {{{.}}},
      {{/recoveredSeries}}
    {{/trimTrailingComma}}
  ]
}

Example: Time Series Information in Output From The Sample Template

Here is a sample alert target output generated with the preceding template:

{
  "failingSeries": [
     ["localhost", "~agent.points.2878.received", ["env=dev","az=us-west-1"]],
     ["db-1", "~agent.points.2878.received", ["env=prod","az=us-west-2"]]
  ],
  "inMaintenanceSeries": [],
  "newlyFailingSeries": [
     ["localhost", "~agent.points.2878.received", ["env=dev","az=us-west-1"]],
     ["db-1", "~agent.points.2878.received", ["env=prod","az=us-west-2"]]
  ],
  "recoveredSeries": []
  }

The template explicitly includes literal text for enclosing the overall list of preformatted strings in square brackets, and for separating the preformatted strings with commas. The trimTrailingComma function suppresses the comma after the last preformatted string. The punctuation (quotation marks, comma separators, and square brackets) in each preformatted output string is generated automatically.

Accessing a Custom Group of Time Series Details

You can access a custom combination of details for the time series that contributed to the alert’s state change. To do so:

  1. Use an alert-series iterator to visit each each time series in the indicated category.
  2. Use variable within the iterator section to access the alert-series details you want to include.

This technique gives you complete control over the formatting of the returned information, and allows you to access statistics from each visited time series.

Alert-Series Iterators

Use the following iterators to visit each time series in the indicated category so you can obtain a custom group of details from each visited time series.

IteratorDefinition
failingAlertSeries Iterator that can return a custom combination of details for each failing time series.
inMaintenanceAlertSeries Iterator that can return a custom combination of details for each time series whose source is in a maintenance window.
newlyFailingAlertSeries Iterator that can return a custom combination of details for each time series that has failed since the previous notification. (These series are also visited by failingAlertSeries.)
recoveredAlertSeries Iterator that can return a custom combination of details for each time series that has recovered since the previous notification.

Alert-Series Details

Use the following variables within the section of an alert-series iterator to specify the details to be included for each visited series. You can use any subset of these variables in any order. Use literal text around these items if you want to format them with any labels, separators, or other punctuation.

VariableDefinition
host Name of the source of the time series being visited.
label Name of the metric of the time series being visited.
tags Iterator that returns a list of the point tags associated with the time series being visited. Each point tag is formatted like this: key=value
observed Number of data points returned by the visited time series during the most recent checking time window.
firing Number of data points in the visited series that are preventing the alert from resolving, during the most recent checking time window. The smaller the number, the closer the series is to recovering. This property is useful only for time series visited by failingAlertSeries and newlyFailingAlertSeries.
stats See Alert-Series Statistics.

Example: Accessing Alert-Series Details in a Generic Webhook Alert Target Template

This portion of the Generic Webhook alert target template shows how to use the failingAlertSeries iterator to retrieve alert-series details for each time series that failed:

"failingAlertSeries": [
  {{#trimTrailingComma}}
    {{#failingAlertSeries}}
      "Source: {{host}}, Label: {{label}}, Tags: {{tags}}, Observed: {{observed}}, Firing: {{firing}}",
    {{/failingAlertSeries}}
  {{/trimTrailingComma}}
]

Example: Alert-Series Details in Output from the Sample Template

The preceding template might yield the following message:

"failingAlertSeries": [
    "Source: raspberrypi, Label: humidity, Tags: {env=production, az=us-west-2}, Observed: 5, Firing: 2"]

Alert-Series Statistics

Statistics provide a profile of the values in a time series during the checking time window immediately preceding a notification. For example, the alert might be set up to fire when a condition is true for 10 minutes. During a 10 minute period where the condition is true, a time series likely have multiple values. You can use statistics to find out, e.g., the largest of these values, or the last value to be reported during the Alert fires time window.

Statistics are normally useful only if you have set the alert’s Display Expression field with a display expression that captures the underlying time series being tested by the condition expression. If the alert has no display expression, statistics are based on the values that are returned by the alert’s condition expression. Because the condition expression returns either 0 or not 0, that information is not useful.

Use the following variables within the section of an alert-series iterator to specify the statistics you want to include for each visited series. You can use any subset of these variables in any order. Use literal text around these items if you want to format them with any punctuation, separators, or labels.

VariableDefinition
stats Set of statistics about the values in the visited time series during the checking time window preceding the notification.
stats.first First value reported within the checking time window immediately preceding the notification.
stats.last Last value reported within the checking time window immediately preceding the notification. <div> Note This value is appended to the output of hostsFailingMessage, which is automatically included in the built-in email and PagerDuty alert targets.</div>
stats.min Minimum value reported within the checking time window immediately preceding the notification.
stats.max Maximum value reported within the checking time window immediately preceding the notification.
stats.mean Average of the values reported within the checking time window immediately preceding the notification.

Example: Accessing Alert-Series Statistics in a Generic Webhook Alert Target Template

This portion of the Generic Webhook alert target template shows how to use the failingAlertSeries iterator to retrieve alert-series statistics for each time series that failed:

"failingAlertSeries": [
  {{#trimTrailingComma}}
    {{#failingAlertSeries}}
      "Source: {{host}}, Label: {{label}}, Tags: {{tags}}, Observed: {{observed}}, Firing: {{firing}},
      First: {{stats.first}}, Last: {{stats.last}}, Min: {{stats.min}}, Max: {{stats.max}}, Mean: {{stats.mean}}",
    {{/failingAlertSeries}}
  {{/trimTrailingComma}}
]

Example: Alert-Series Statistics in Output from the Sample Template

The preceding template might yield the following message:

"failingAlertSeries": [
    "Source: raspberrypi, Label: humidity, Tags: {env=production, az=us-west-2}, Observed: 5, Firing: 2,
    First: 46.6, Last: 46.0, Min: 46.0, Max: 46.6, Mean: 46.279999999999994"]

Tailoring Content to the Trigger Type

If you want to send out different notifications for different types of triggers, you can use the following functions. For example, you can use the same template to send out one message for a firing alert, and another message for an updated alert.

FunctionDefinition
isAlertOpened Includes the contents of the section only if the alert has transitioned from checking to firing.
isAlertUpdated Includes the contents of the section only if the alert is updated - that is, if the alert continues firing while an individual time series changes from recovered to failing, or from failing to recovered.
isAlertResolved Includes the contents of the section only if the alert is fully resolved, and so has transitioned from firing to checking.
isAlertMaintenance Includes the contents of the section only if one or more sources associated with the alert are in an ongoing maintenance window.
isAlertSnoozed Includes the contents of the section only if the alert has been snoozed.

Example: Conditionalizing Content in a Plain Text Alert Target Template

Here is an alert target template for plain text notifications that sends the text “Alert is firing!” if the alert opened, and sends different messages if the alert is updated or resolved.

{{! Alert Opened section }}
{{#isAlertOpened}}
  Alert is firing!
{{/isAlertOpened}}

{{! Alert Updated section }}
{{#isAlertUpdated}}
  An individual time series failed or recovered while at least one other time series is firing!
{{/isAlertUpdated}}

{{! Alert Resolved section }}
{{#isAlertResolved}}
  Alert has resolved!
{{/isAlertResolved}}

Here is the output in a notification that was triggered by the alert firing.

Alert is firing!

Limiting List Sizes

If your messaging platform imposes a limit on the number of characters in a notification, you can avoid exceeding this limit by setting a limit on the number of items returned by iterators.

The default value for each limit you can set with a customization function is 500. You must set the limit before iteration or the limit does not take effect.

The order of the limit settings determines limit precedence. For example, if you first set setDefaultIterationLimit and then you set setFailingLimit, then setFailingLimit overwrites the setDefaultIterationLimit setting.

The failingLimit property applies to all iterators in the failing category: failingAlertSeries, failingSeries, and failingHosts.

See Setting and Testing Iteration Limits for an example.

FunctionDefinition
getIterationLimit Gets the value of an iteration limit. Valid values are: defaultIterationLimit, failingLimit, inMaintenanceLimit, newlyFailingLimit, and recoveredLimit.
iterationLimitExceed Checks whether the number of the result returned is limited by an iteration limit. Valid values are: failingLimitExceed, inMaintenanceLimitExceed, newlyFailingLimitExceed, and recoveredLimitExceed.
setDefaultIterationLimit Sets all limits to the same value.
setFailingLimit Sets the limit for the number of items returned by failingAlertSeries, failingHosts, and failingSeries.
setInMaintenanceLimit Sets the limit for the number of items returned by inMaintenanceAlertSeries, inMaintenanceHosts, and inMaintenanceSeries.
setNewlyFailingLimit Sets the limit for the number of items returned by newlyFailingAlertSeries, newlyFailingHosts, and newlyFailingSeries.
setRecoveredLimit Sets the limit for the number of items returned by recoveredAlertSeries, recoveredHosts, and recoveredSeries.

Example: Setting and Testing Iteration Limits

Suppose you have 8 failing sources: source1, source2, source3, source4, source5, source6, source7, source8. You set setDefaultIterationLimit to 5 in the first line of the following template:

{{#setDefaultIterationLimit}}5{{/setDefaultIterationLimit}}
{
  "getIterationLimit": {
     "defaultIterationLimit": "{{{defaultIterationLimit}}}",
     "failingLimit": "{{{failingLimit}}}",
     "inMaintenanceLimit": "{{{inMaintenanceLimit}}}",
     "newlyFailingLimit": "{{{newlyFailingLimit}}}",
     "recoveredLimit": "{{{recoveredLimit}}}"
   },
   "iterationLimitExceed": {
     "failingLimitExceed": "{{{failingLimitExceed}}}",
     "inMaintenanceLimitExceed": "{{{inMaintenanceLimitExceed}}}",
     "newlyFailingLimitExceed": ""{{{newlyFailingLimitExceed}}}",
     "recoveredLimitExceed": "{{{recoveredLimitExceed}}}"
   },
  "alertId": "{{{alertId}}}",
  "alertTags": [
    {{#trimTrailingComma}}
      {{#alertTags}}
        "{{#jsonEscape}}{{{.}}}{{/jsonEscape}}",
      {{/alertTags}}
    {{/trimTrailingComma}}
  ],
  ...
  "failingSources": [
    {{#trimTrailingComma}}
      {{#failingHosts}}
        "{{{.}}}",
      {{/failingHosts}}
    {{/trimTrailingComma}}
  ]
}

The template with these settings produces the following output for the 8 failing sources:

{
 "getIterationLimit": {
   "defaultIterationLimit": "5",
   "failingLimit": "5",
   "inMaintenanceLimit": "5",
   "newlyFailingLimit": "5",
   "recoveredLimit": "5"
 },
 "iterationLimitExceed": {
   "failingLimitExceed": "true",
   "inMaintenanceLimitExceed": "false",
   "newlyFailingLimitExceed": "false",
   "recoveredLimitExceed": "false"
 },
 "alertId": "1492543979795",
 "alertTags": ["production", "mysql"],
 ...
 "failingSources": ["source5", "source4", "source7", "source6", "source1"]
}

failingHosts iterates only up to failingLimit, which is 5 in this case. failingLimitExceed is true because the number of failing sources exceeds the limit.

In contrast, if the failingLimit is 10, the output is the following for 8 failing sources:

{
  "getIterationLimit": {
    "defaultIterationLimit": "10",
    "failingLimit": "10",
    "inMaintenanceLimit": "10",
    "newlyFailingLimit": "10",
    "recoveredLimit": "10"
  },
  "iterationLimitExceed": {
    "failingLimitExceed": "false",
    "inMaintenanceLimitExceed": "false",
    "newlyFailingLimitExceed": "false",
    "recoveredLimitExceed": "false"
  },
  "alertId": "1492543979795",
  "alertTags": [production, mysql],
  ...
  "failingSources": ["source5", "source4", "source7", "source6", "source1", "source3", "source2", "source8"]
}

For this case (limit 10, failing sources 8) failingLimitExceed is false because the number of failing sources does not exceed the limit set.

Utility Functions for Readability

You can use alert target utility functions to make the output of the alert target more readable.

  • Use jsonEscape if you send notifications to a messaging platform that uses JSON.
  • Use xml11Escape or xml10Escape if you send notifications to a messaging platform that uses XML.
  • Use trimTrailingComma if you send notifications to a messaging platform that does not automatically suppress a literal comma after the final element of a list.
FunctionDefinitionExample
jsonEscape Escapes the characters in a string using JSON string rules. Escapes any values it finds into their Json string form. Deals correctly with quotes and control-chars (tab, backslash, cr, ff, etc) so, for example, a tab becomes the characters \\ and t. Input: He didn't say, "Stop!"
Output: He didn't say, \"Stop!\"
xml11Escape
Escapes the characters in a String using XML entities. XML 1.1 can represent certain control characters, but it cannot represent the null byte or unpaired Unicode surrogate codepoints, even after escaping.
escapeXml11 removes characters that do not fit in the following ranges:
[#x1-#xD7FF]|[#xE000-#xFFFD]|[#x10000-#x10FFFF]
escapeXml11 escapes characters in the following ranges:
[#x1-#x8]|[#xB-#xC]|[#xE-#x1F]|[#x7F-#x84]|[#x86-#x9F]
Input: "bread" & "chocolate"
Output: &quot;bread&quot; &amp; &quot;chocolate&quot;
xml10Escape
Escapes the characters in a string using XML entities. XML 1.0 is a text-only format, it cannot represent control characters or unpaired Unicode surrogate codepoints, even after escaping.
escapeXml10 removes characters that do not fit in the following ranges: #x9|#xA|#xD|[#x20-#xD7FF]|[#xE000-#xFFFD]|[#x10000-#x10FFFF]
escapeXml10 escapes characters in the following ranges: [#x7F-#x84]|[#x86-#x9F]
Input: "bread" & "chocolate"
Output: &quot;bread&quot; &amp; &quot;chocolate&quot;
trimTrailingComma Retains the string content up to the last comma. Often used within iterator sections to remove the extra comma after the last element visited by an iterator. Input:
"(Host: "xyz", Label: 3.0), "
Output:
"(Host: "xyz", Label: 3.0) "

Adding Chart Images to Older Custom Alert Targets

As of 2018-26.x, the predefined template for a custom HTML email target or a custom Slack target automatically includes the imageLinks variable for producing a chart image in alert notifications. However, if you created a custom email alert target or a custom Slack alert target before 2018-26.x, you must explicitly update the alert target’s template to include a chart image in the alert notifications.

Note You do not need to update pre-existing custom alert targets of type PagerDuty. All PagerDuty notifications sent in 2018-26.x or later will include chart images.

Updating a Pre-Existing Custom Email Alert Target

To update a custom email alert target that was created before 2018-26.x:

  1. Open the custom alert target for editing.
  2. Select HTML Format. (All custom email alert targets created before 2018-26.x are plain text.)
  3. In the Body Template box, insert a snippet such as the following:

     {{#imageLinks}}
     <img src="{{{.}}}" />
     {{/imageLinks}}
    

Subsequent email notifications will now include a chart image that is generated for the alert. (Without the HTML <img src= > tag, the value returned by the imageLinks iterator would be displayed as a URL to a chart image, and not as an image.)

Updating a Pre-Existing Custom Alert Target for Slack

To update the template for a custom Slack alert target that was created before 2018-26.x:

  1. Open the custom alert target for editing.
  2. In a separate browser tab, connect to your Wavefront service and create a new custom alert target of type Webhook with the Slack template.
  3. In the newly created template, find and copy the section enclosed in the following lines:

     {{#imageLinks}}
       ...
     {{/imageLinks}}
    
  4. Paste the copied section into template of the pre-existing alert target.