Learn how to configure Wavefront proxy preprocessor rules.

Starting with version 4.1, the Wavefront proxy includes a preprocessor that applies rules before data is sent to Wavefront. We support:

  • Point filtering rules
  • Point altering rules
  • Span filtering rules (new in proxy 4.38)
  • Span altering rules (new in proxy 4.38)

The rules make it possible to address data quality issues in the data flow when it’s not possible to fix the problem at the source. For example, you could have a rule “before the point line is parsed, replace invalid characters with underscores” to allow points that would be rejected to get to the Wavefront service.

Rule Configuration File

You define the proxy preprocessor rules in a rule configuration file, usually <wavefront_config_path>/preprocessor_rules.yaml, using YAML syntax. You can specify rule filenames in your proxy configuration. An example rule file could look like this:

# rules for port 2878
'2878':
  # replace bad characters ("&", "$", "!", "@") with underscores in the entire point line string
  ################################################################
  - rule    : replace-badchars
    action  : replaceRegex
    scope   : pointLine
    search  : "[&\\$!@]"
    replace : "_"

  #  remove "az" point tag if its value starts with "dev"
  ################################################################
  - rule    : drop-az-tag
    action  : dropTag
    tag     : az
    match   : dev.*

# rules for port 4242
'4242':
  #  remove "az" point tag if its value starts with "dev"
  ################################################################
  - rule    : drop-az-tag
    action  : dropTag
    tag     : az
    match   : dev.*

  ...

You can define separate rules for each listening port. The example above defines 2 rules for port 2878 and 1 rule for port 4242.

Rule Parameters

Every rule must have

  • A rule parameter that contains the rule ID. Rule IDs can contain alphanumeric characters, dashes, and underscores and should be descriptive and unique within the same port. In the example above, the drop-az-tag rule is defined with the same identifier for both ports, 2878 and 4242.
  • An action parameter that contains the action to perform. Also the rule name.

Additional parameters depend on the rule that you’re defining, for example, a whitelistregex rule must have a scope and a match parameter.

Regex Notes

  • Backslashes in regex patterns must be double-escaped. For example, to match a dot character (“.”), use \\..
  • Regex patterns in the match parameter are a full match. For example, a regex to block the point line that contains stage substring is .*stage.*.
  • Regex patterns in the replaceRegex rule search parameter are a substring match. If search is “A” and replace is “B”, all A’s are replaced with B’s.

Enabling the Preprocessor

To enable the preprocessor:

  1. Add (or uncomment) the preprocessorConfigFile property in the Wavefront proxy configuration file.
  2. Set preprocessorConfigFile to a valid path to the rules configuration file.

Validation and Changes to the Preprocessor File

The rules file is validated when the proxy starts. The proxy aborts the start-up process if any of the rules is not valid. We provide a detailed error message for every rule that fails validation.

Starting with proxy version 5.0, changes to the preprocessor file take effect shortly after you save the file.

For earlier versions of the proxy, you have to restart the proxy before the changes take effect.

Metrics for Rules

For every rule the Wavefront proxy reports a counter metric that shows how often the rule has been successfully applied. The rule ID becomes part of the proxy metric ~agent.preprocessor.<ruleID>.count, for example, ~agent.preprocessor.replace-badchars.count. See Monitoring Wavefront Proxies for details.

Point Filtering Rules

Point filtering rules support a more flexible version of the proxy whitelistRegex and blacklistRegex properties, and is fully backwards compatible.

blacklistRegex

Defines a regex that points must match to be filtered out.

Parameters
Parameter Description
action blacklistRegex.
scope Any of the following:
  • pointLine—Rule applies to the whole point line before it's parsed (can be used with Wavefront and Graphite formats only).
  • metricName—Rule applies only to the metric name after the point is parsed.
  • sourceName—Rule applies only to the source name after the point is parsed.
  • <point tag>—For any other value, the rule applies to the value of a point tag with this name after the point is parsed.
match A regex pattern that input lines must match to be filtered out.
Examples
  # block all points with sourceName that starts with qa-statsd
  ###############################################################
  - rule    : example-block-qa-statsd
    action  : blacklistRegex
    scope   : sourceName
    match   : "qa-statsd.*"

  # block all points where "datacenter" point tag value starts with "west"
  ###############################################################
  - rule    : example-block-west
    action  : blacklistRegex
    scope   : datacenter
    match   : "west.*"

whitelistRegex

Points must match the whitelistRegex to be accepted. Multiple whitelistRegex rules are allowed. A point must match all rules.

Parameters
Parameter Description
action whitelistRegex.
scope Any of the following:
  • pointLine—Rule applies to the whole point line before it's parsed (can be used with Wavefront and Graphite formats only).
  • metricName—Rule applies only to the metric name after the point is parsed.
  • sourceName—Rule applies only to the source name after the point is parsed.
  • <point tag>—For any other value, the rule applies to the value of a point tag with this name after the point is parsed.
match A regex pattern that input lines must match to be accepted.
Examples
  # only allow points that contain "prod" substring anywhere in the point line
  ###############################################################
  - rule    : example-allow-only-prod
    action  : whitelistRegex
    scope   : pointLine
    match   : ".*prod.*"

  # only allow points that have a "datacenter" point tag and its value starts with "west"
  ###############################################################
  - rule    : example-allow-only-west
    action  : whitelistRegex
    scope   : datacenter
    match   : "west.*"

Point Altering Rules

Point altering rules allow you to:

  • Replace text in the point line. The point line uses the Wavefront data format input format: <metricName> <metricValue> [<timestamp>] source=<source> [pointTags]
  • Add, remove, or update point tags.

replaceRegex

Replaces arbitrary text in the point line or any of its components:

Parameters
Parameter Description
action replaceRegex.
scope Any of the following:
  • pointLine—Rule applies to the whole point line before it's parsed (can be used with Wavefront and Graphite formats only).
  • metricName—Rule applies only to the metric name after the point is parsed.
  • sourceName—Rule applies only to the source name after the point is parsed.
  • <point tag>—For any other value, the rule applies to the value of a point tag with this name after the point is parsed.
search Search pattern. All substrings that match this pattern are replaced with the replacement string.
replace Replacement string. The empty string is allowed. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
match (optional) Regular expression. If specified, extract the tag only if the point line, source name, metric name or point tag value matches this regular expression.
Examples
# for "exampleCluster" point tag replace all "-" characters with dots
###############################################################
- rule    : example-cluster-name
  action  : replaceRegex
  scope   : exampleCluster
  search  : "-"
  replace : "."

# replace bad characters ("&", "$", "!") with underscores in the entire point line string
################################################################
- rule    : example-replace-badchars
  action  : replaceRegex
  scope   : pointLine
  search  : "[&\\$!]"
  replace : "_"

addTag and addTagIfNotExists

Adds a point tag with the specified value.

  • For addTag, if the point tag already exists, its existing value is replaced with the new value.
  • For addTagIfNotExists, if the point tag already exists, its existing value is preserved.
Parameters
Parameter Description
action addTag
addTagIfNotExists
tag New point tag name.
value New point tag value.
Example
  # add "env=prod" point tag to all metrics sent through this port
  ################################################################
  - rule    : tag-all-metrics
    action  : addTag
    tag     : env
    value   : "prod"

    # add "env=prod" point tag to all metrics sent through this port unless already tagged with "env"
    ################################################################
    - rule    : tag-all-metrics
      action  : addTagIfNotExists
      tag     : env
      value   : "prod"

dropTag

Removes a point tag.

Parameters
Parameter Description
action dropTag.
tag Point tag key (or a regex matching the tag key).
match (optional) Regular expression. If specified, remove a tag only if its value matches this regular expression.
Examples
  #  remove "dc" point tag from all points
  ################################################################
  - rule    : drop-dc-tag
    action  : dropTag
    tag     : dc

  #  remove "az" point tag if its value starts with "dev"
  ################################################################
  - rule    : drop-az-tag
    action  : dropTag
    tag     : az
    match   : dev.*

extractTag and extractTagIfNotExists

Extracts a string from the metric name, source name, or point tag value and creates a new point tag from it.

  • For extractTag create the new point tag.
  • For extractTagIfNotExists create the new point tag but do not replace the existing value with the new value if the tag already exists.
Parameters
Parameter Description
action extractTag
extractTagIfNotExists
source Any of the following:
  • pointLine—Rule applies to the whole point line before it's parsed (can be used with Wavefront and Graphite formats only).
  • metricName—Rule applies only to the metric name after the point is parsed.
  • sourceName—Rule applies only to the source name after the point is parsed.
  • <point tag>—For any other value, the rule applies to the value of a point tag with this name after the point is parsed.
tag Name of the new point tag.
search Regex pattern for the value to replace.
replace Replacement string or pattern that will be used as a value for the new point tag. Empty string is allowed. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
match (optional) Regular expression. If specified, extract a tag only if "source" (source name, metric name or point tag value) matches this regular expression.
Example
  # extract a "datacenter" point tag from the source name based on '.dc-' substring.
  # it will extract datacenter=west01 tag from source host0001.web.dc-west01.corp
  ####################################################################################
  - rule    : extract-datacenter
    action  : extractTag
    source  : sourceName
    tag     : datacenter
    search  : "^.*\\.dc-(.*)\\..*"
    replace : "$1"

renameTag

Renames a point tag, preserving its value.

Parameters
Parameter Description
action renameTag
tag Point tag to be renamed.
newtag New name for the point tag.
match (optional) Regular expression. If specified, rename a tag if its value matches this regular expression.
Examples
  # rename a "dc" point tag to "datacenter" (unconditional)
  ###############################################################
  - rule    : rename-dc-to-datacenter
    action  : renameTag
    tag     : dc
    newtag  : datacenter

  # rename a point tag if its value is numeric. so oldTag=123 would be renamed to numericTag=123, but oldTag=text123 would not be changed.
  ###############################################################
  - rule    : rename-numeric-tag
    action  : renameTag
    tag     : oldTag
    match   : "^\\d*$"
    newtag  : numericTag

forceLowercase

Converts metric name, source name, or point tag value to lowercase.

Parameters
Parameter Description
action forceLowercase
scope Any of the following:
  • pointLine—Rule applies to the whole point line before it's parsed (can be used with Wavefront and Graphite formats only).
  • metricName—Rule applies only to the metric name after the point is parsed.
  • sourceName—Rule applies only to the source name after the point is parsed.
  • <point tag>—For any other value, the rule applies to the value of a point tag with this name after the point is parsed.
match (optional) Regular expression. If specified, convert a tag to lower case only if its value matches this regular expression.
Example
  # force lowercase on metric name (with optional regex match)
  ################################################################
  - rule    : example-force-lowercase
    action  : forceLowercase
    scope   : metricName
    match   : "^UPPERCASE.*$"

limitLength

Enforces string length limits for a metric name, source name, or point tag value. Available action sub-types are truncate, truncateWithEllipsis, and drop.

Parameters
Parameter Description
action limitLength. Required.
scope Any of the following:
  • pointLine—Rule applies to the whole point line before it's parsed (can be used with Wavefront and Graphite formats only).
  • metricName—Rule applies only to the metric name after the point is parsed.
  • sourceName—Rule applies only to the source name after the point is parsed.
  • <point tag>—For any other value, the rule applies to the value of a point tag with this name after the point is parsed.
actionSubtype Allows you determine how we limit length:
  • drop—Drops requested scope if value is greater than maxLength. You can't use DROP with the metric name or source name.
  • truncate—Truncates requested scope if value is greater than maxLength.
  • truncateWithEllipsis—Truncates the requested scope if the value is greater than maxLength but preserving ellipsis (three dots). maxLength must be at least 3 for this action type.
maxLength The maximum length of the input metric name, source name, or point tag. Only if the length of the input is greater than maxLength, the rule is applied.
match (optional) Regular expression. If specified, limitLength applies only to tags with a value that matches this regular expression.RK>>Tags with a value or tags??
Example
# Truncate the length of all metric names starting with  "metric"
# i.e. from "metric.name.2.test" to "metric.2.name..."
################################################################
- rule : limit-metric-name-length
  action : limitLength
  scope : metricName
  actionSubtype : truncateWithEllipsis
  maxLength : 16
  match : "^metric.*"

Span Filtering Rules

Wavefront distributed tracing gives you end-to-end visibility into an entire request across services by allowing you to examine traces and spans. Span filtering rules allow you to specify a black list or white list that determine which spans the proxy sends to the Wavefront service.

spanBlacklistRegex

Defines a regex that spans must match to be filtered out. In the example below, we don’t allow spans with a source name that starts with qa-service.

Parameters
Parameter Description
action blacklistRegex
scope Any of the following:
  • spanName—Rule applies only to span name.
  • sourceName—Rule applies only to the source name.
  • <span tag>—For any other value, the rule applies to span tags (annotations).
match A regex pattern. If the input matches this regex, it is filtered out.
Examples
# block all spans with sourceName that starts with qa-service
###############################################################
- rule    : example-span-block-qa-services
  action  : spanBlacklistRegex
  scope   : sourceName
  match   : "qa-service.*"

spanWhitelistRegex

Points must match the spanWhitelistRegex to be accepted. Multiple spanWhitelistRegex rules are allowed. A point must match all rules.

Parameters
Parameter Description
action spanWhitelistRegex.
scope Any of the following:
  • spanName—Rule applies only to the span name.
  • sourceName—Rule applies only to the source name.
  • <span tag>—For any other value, the rule applies to span tags (annotations).
match A regex pattern that input lines must match to be accepted.
Examples
# only allow spans that contain the "prod" substring anywhere in the source
###############################################################
- rule    : example-span-allow-only-prod
  action  : spanWhitelistRegex
  scope   : sourceName
  match   : ".*prod.*"

Span Altering Rules

Span altering rules allow you to add, remove, or update span tags.

spanReplaceRegex

Replaces arbitrary text in the span name, span source name, or a span tag.

Parameters
Parameter Description
action spanReplaceRegex
scope Any of the following:
  • spanName—Rule applies only to the span name.
  • sourceName—Rule applies only to the source name.
  • <point tag>—For any other value, the rule applies to span tags (annotations).
search Search pattern. All substrings matching this pattern are replaced with the replacement string.
replace Replacement string. The empty string is allowed. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
match (optional) Regular expression. If specified, extract the tag only if the span name, source name, or span tag matches this regular expression.
iterations (optional) Number of iterations. Recursively check and recursively replace if the output string contains the search string until the number of iterations is reached.
firstMatchOnly If set to true, performs string replacement only on the first matching span tag’s value. Only applicable when scope is a span tag. Default is false.
Examples
# replace dashes with with underscores in the span name
# if span name starts with "app", i.e. change "app.span-service-frontend" to
# "app.span_service_frontend"
################################################################
- rule    : example-app-span-replace-dashes
  action  : spanReplaceRegex
  scope   : spanName
  search  : "-"
  replace : "_"
  match   : "^app.*"

spanForceLowercase

Convert a span name, source name, or span tag name to lowercase.

Parameters
Parameter Description
action spanForceLowercase
scope Any of the following:
  • spanName—Rule applies only to the span name.
  • sourceName—Rule applies only to the source name.
  • <span tag>—For any other value, the rule applies to span tags (annotations).
match (optional) Regular expression. If specified, force lower case only if the value matches this regular expression.
firstMatchOnly If set to true, performs string replacement only on the first matching span tag’s value. Only applicable when scope is a span tag. Default is false.
Example
# force lowercase on span name (with optional regex match)
################################################################
- rule    : example-span-force-lowercase
  action  : spanForceLowercase
  scope : spanName
  match : "^UPPERCASE.*$"
  firstMatchOnly : false

spanAddTag and spanAddTagIfNotExists

Add a span tag to all spans.

  • For spanAddTag, if the tag already exists, its existing value is replaced with the new value.
  • For spanAddTagIfNotExists, do not replace the value of an existing tag.
Parameters
Parameter Description
action spanAddTag
spanAddTagIfNotExists
key New span tag name.
value New span tag value.
Example
  # add "env=prod" point tag to all spans sent through this port
  ################################################################
  - rule    : example-span-tag-all-metrics
    action  : spanAddTag
    tag     : env
    value   : "prod"

  # add "env=prod" point tag to all spans sent through this port
  # unless already tagged with "env"
  ################################################################
  - rule    : example-span-tag-all-metrics-if-not-exists
    action  : spanAddTagIfNotExists
    tag     : env
    value   : "prod"

spanDropTag

Removes a span tag that matches a regex string.

Parameters
Parameter Description
action spanDropTag
key Span tag name (or a regex matching the tag name).
match (optional) If specified, remove a tag only if its value matches this regular expression.
firstMatchOnly If set to true, removes only the first matching tag. Default is false.
Examples
#  remove "az" annotation if its value starts with "dev"
################################################################
- rule    : example-span-drop-az-annotation
  action  : spanDropTag
  tag     : az
  match   : dev.*
  firstMatchOnly : false

# remove first "bindTo" tag where a value starts with "dev"
# i.e. `bindTo=prod1 bindTo=dev1 bindTo=prod2 bindTo=dev2` will become
# `bindTo=prod1 bindTo=prod2 bindTo=dev2`
################################################################
- rule           : example-span-drop-bindto-dev-tags
  action         : spanDropTag
  tag            : bindTo
  match          : ^dev.*
  firstMatchOnly : true

spanExtractTag and spanExtractTagIfNotExists

Extract a string from a span name, source name, or a span tag value and create a new span tag from that string.

  • For spanExtractTag, create the new span tag.
  • For spanExtractTagIfNotExists, do not create the new span tag if at least one tag with this name already exists.
Parameters
Parameter Description
action spanExtractTag
spanExtractTagIfNotExists
key New tag name.
input Any of the following:
  • spanName—Rule applies only to the span name.
  • sourceName—Rule applies only to the source name.
  • <span tag>—For any other value, the rule applies to span tags (annotations).
match (optional) Regular expression. If specified, extract a tag only if the input (source name, span name or span tag value) matches this regular expression.
search Regex pattern to extract the value from.
replace String or pattern that will be used as a value for the new span tag. Empty string is allowed. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
replaceInput (Optional) Modify the name of the input. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
firstMatchOnly If set to true, performs string replacement only on the first matching span tag’s value. Only applicable when input is a span tag and replaceInput is specified. Default is false.
Examples
## extract 3rd dot-delimited node from the span name into new
## span tag, and remove it from the metric, i.e. from
## "span.service.frontend.cpu_utilization" span name extract "serviceTag=frontend"
## tag and change metric name to "span.service.cpu_utilization"
################################################################
- rule          : example-extract-tag-from-span
  action        : spanExtractTag
  key           : serviceTag
  input         : spanName
  match         : "span.*"
  search        : "^([^\\.]*\\.[^\\.]*\\.)([^\\.]*)\\.(.*)$"
  replace       : "$2"
  replaceInput  : "$1$3"
  # optional, omit if you plan on just extracting the tag leaving the metric name intact
## extract 3rd dot-delimited node from the span name into new
## span tag, and remove it from the metric, i.e. from
## "span.service.frontend.cpu_utilization" span name extract "serviceTag=frontend"
## tag and change metric name to "span.service.cpu_utilization"
################################################################
- rule          : example-extract-tag-from-span
  action        : spanExtractTagIfNotExists
  key           : serviceTag
  input         : spanName
  match         : "span.*"
  search        : "^([^\\.]*\\.[^\\.]*\\.)([^\\.]*)\\.(.*)$"
  replace       : "$2"
  replaceInput  : "$1$3"
  # optional, omit if you plan on just extracting the tag leaving the metric name intact

spanlimitLength

Truncate the span name or source name to the given length. Truncate or drop span tags if tag value length exceeds the limit.

Available action subtypes are truncate, truncateWithEllipsis, and drop.

Parameters
Parameter Description
action spanlimitLength
scope Any of the following:
  • spanName—Rule applies only to the span name of the span.
  • sourceName—Rule applies only to the source name of the span.
  • <span tag>—For any other value, the rule applies to span tags (annotations).
actionSubtype Allows you determine how we limit length:
  • drop—Drops requested scope if value is greater than maxLength. You can't use DROP with the metric name or source name.
  • truncate—Truncates requested scope if value is greater than maxLength.
  • truncateWithEllipsis—Truncates the requested scope if the value is greater than maxLength but preserving ellipsis (three dots). maxLength must be at least 3 for this action type.
maxLength The maximum length of the input span name, source name, or span tag. The length of the input must be greater than maxLength for rule to be applied.
match (optional) Regular expression. If specified, remove a tag if its value matches this regular expression.
firstMatchOnly If set to true, applies only to the first matching span tag. Only applicable when the scope is a span tag. Default is false.
Example
## truncate 'db.statement' annotation value at 240 characters,
## replace last 3 characters with '...'.
################################################################
- rule          : example-limit-db-statement
  action        : spanLimitLength
  scope         : "db.statement"
  actionSubtype : truncateWithEllipsis
  maxLength     : "240"