Proxy configuration properties and preprocessor rules for logging.

Proxy configuration properties let you customize proxy behavior. Proxy preprocessor rules let you block, allow, or replace parts of the incoming traffic. This doc page is a reference to properties specific to the Logs feature. See Advanced Proxy Configuration and Proxy Preprocessor Rules for the properties and rule options for other kind of data.

Proxy Configuration Properties for Logs

If your logs don’t use the attributes that our logging solution expects, or if you want to customize proxy behavior otherwise, you can use proxy configuration properties to do that.

We’ve added the following configuration properties for logs to the already existing proxy configuration properties.

Properties for Changing Log Tags

Property Purpose Format /Example
customTimestampTags Comma-separated list of log tag keys that are treated as the timestamp if your logging solution doesn't send a timestamp or log_timestamp tag.
Default: None.
Version: Since 11.3		 Comma-separated list of tags. Can be a single tag.
Example: sent_time
customMessageTags Comma-separated list of log tag keys that are treated as the log message if your logging solution doesn’t send a message or text tag.
Default: None.
Version: Since 11.3		 Comma-separated list of tags. Can be a single tag.
Example: debug_log
customApplicationTags Comma-separated list of log tag keys that are treated as the application name if your logging solution doesn’t send an application tag.
Default: none
Version: Since 11.3		 Comma separated list of log tags. Can be a single tag.
Example: supermarket
customServiceTags Comma-separated list of log tag keys that are treated as the service name if your logging solution doesn’t send a service tag.
Default: none
Version: Since 11.3		 Comma separated list of tags. Can be a single tag.
Example: groceries, payment
customExceptionTags Comma separated list of log tag keys that are treated as the exception if the logging solution doesn’t send a tag named exception.
Default: exception, error_name
Version: Since 11.3		 Comma separated list of tags. Can be a single tag.
Example: warn, error
customLevelTags Comma separated list of log tag keys that are treated as the log level if the logging solution doesn’t send a tag named level.
Default: level, log_level
Version: Since 11.3		 Comma separated list of tags. Can be a single tag.
Example: log_level

Properties for Modifying Proxy Behavior

Property Purpose Format /Example
pushRateLimitLogs Limit the outgoing logs rate at the proxy, in MB/s.
Default: NO_RATE_LIMIT.
Version: Since 11.3		 Positive integer.
Example: 1
pushFlushIntervalLogs Milliseconds between sending batches of logs.
Default: 1000.
Version: Since 11.3		 Number of milliseconds.
Example: 2000
pushFlushMaxLogs The maximum size of the log payload that is sent to the server in a single flush. The value needs to be in Bytes. If the log exceeds the value mentioned here, the log is not sent to the server.
Default: 4194304 (4 MB)
Version: Since 12.0		 A value between 1048576 and 5242880 (1 MB and 5 MB).
Example: 3145728
flushThreadsLogs Number of threads that flush data to the server. This setting is per push listener port.
If you set a large value, for example 10, then the number of logs that are included in a batch will be small and sending logs will be expensive (e.g. CPU-intensive) because you need to connect to the server several times.
Default: 4
Version: Since 11.3		 Positive integer.
Example: 5
pushMemoryBufferLimitLogs Maximum number of logs that can stay in the proxy memory buffer before spooling to disk. Change this property based on how much memory capacity for the proxy your system has.
  • If the value is higher, the proxy is better able to handle surges.
  • If the value is lower, it reduces memory usage but will force the proxy to spool to disk more frequently when the logs data points arrive at the proxy in short bursts.
  • If the limit is the same as the memory heap limit, the proxy ignores the limit you have set here.

Default: 64
Version: Since 11.3		 Positive integer.
Example: 70
blockedLogsLoggerName Logger name for blocked logs in the proxy log.
Default: RawBlockedLogs
Version: Since 11.3		 A string.
Example: blockedLogs

Proxy Preprocessor Rules for Logs

The Wavefront proxy includes a preprocessor that applies rules before the log data is sent to our service. Logs store data in tags, that are key-value pairs. The rules listed below, update the log tag value.

logReplaceRegex

Replaces content in the tag value.

Parameters
Parameter Description
action logReplaceRegex
scope Rule applies to the value of the specified log tag key
search Search pattern. All substrings matching this pattern are replaced with the replacement string.
replace Replacement string. The empty string is allowed. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
match (optional) Regular expression. If specified, extract the tag only if the log tag matches this regular expression.
iterations (optional) Number of iterations. Recursively check and recursively replace if the output string contains the search string until the number of iterations is reached.
firstMatchOnly (optional) If set to true, performs string replacement only on the first matching log tag’s value. Only applicable when scope is a log tag. Default is false.
Examples 
# searches for the word foo on the tag value if the tag key is message, and replaces the word foo with bar.
- rule          : test-logreplaceregex
  action        : logReplaceRegex
  scope         : message
  search        : foo
  replace       : bar

logForceLowercase

Converts the tag key to lowercase.

Parameters
Parameter Description
action logForceLowercase
scope Rule applies to the value of the specified log tag key.
match (optional) Regular expression. If specified, convert a tag to lower case only if its value matches this regular expression.
Example 

# forces the sourceName tag key to lowercase: (sourcename)
 - rule          : test-logforcelowercase
   action        : logForceLowercase
   scope         : sourceName

logAddTag and logAddTagIfNotExists

Add a log tag to all logs.

  • logAddTag adds the new log tag and assigns the new value to it. If you want to update the value of a log tag, you need to drop the log tag and add it again.
  • logAddTagIfNotExists adds the log tag only if it does not already exist.
Parameters
Parameter Description
action logAddTag
logAddTagIfNotExists
key New log tag name.
value New log tag value.
Example 
# adds customTag1:val1 to all the log data.
- rule          : test-logaddannotation
  action        : logAddTag
  key           : customTag1
  value         : "val1"

################################################################

# adds customTag2:val2 if customTag1 does not already exist
# this rule will not be active because customTag1 was added in previous rule
- rule          : test-logaddTagifnotexists
 action        : logAddTagIfNotExists
 key           : customTag1
 value         : "val2"

logDropTag

Removes a log tag that matches a regex string.

Parameters
Parameter Description
action logDropTag
key Log tag name (or a regex matching the tag name).
match (optional) If specified, remove a tag only if its value matches this regular expression.
Examples 
# drops the datacenter tag if the value matches az4, az5, az6.
 - rule          : test-logDropTag
   action        : logDropTag
   key           : datacenter
   match         : "az[4-6]"

logExtractTag and logExtractTagIfNotExists

Extract a string from a log tag name, or a tag tag value and create a new log tag from that string.

  • For logExtractTag, create the new log tag.
  • For logExtractTagIfNotExists, do not create the new log tag if at least one tag with this name already exists.
Parameters
Parameter Description
action logExtractTag
logExtractTagIfNotExists
key New tag name.
input Rule applies to the value of the specified log tag (annotation) key.</li> </ul>
match (optional) Regular expression. If specified, extract a tag only if the log tag value matches this regular expression.
search Regex pattern to extract the value from.
replace String or pattern that will be used as a value for the new log tag. An empty string is allowed. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
replaceInput (optional) Modify the name of the input. Refer to a capturing group in the search regex using $ and its number (starting from 1). For example, use $1 to refer to the first group.
Examples 
# turns tagtoExtract:foobar to tagToExtract:fooar and extractedTag:b
 - rule          : test-logExtractAnnotation
   action        : logExtractAnnotation
   key           : extractedTag
   input         : tagToExtract
   search        : "(foo)(b)(ar)"
   replace       : "$2"
   replaceInput  : "$1$3"


# same as logExtractAnnotation. if tagToExtract already exist, this rule never runs because the tag is already there.
- rule          : test-logextracttagifnotexists
  action        : logExtractAnnotationIfNotExists
  key           : extractedTag
  input         : tagToExtract
  search        : "(foo)(b)(ar)"
  replace       : "$1"
  replaceInput  : "$2$3"

logRenameTag

Renames a log tag. The renaming does not affect the values stored in a log.

Parameters
Parameter Description
action logRenameTag
key The log tag to be renamed.
newkey The new name for the log tag.
match (optional) If specified, renames a log tag if its value matches this regular expression.
Example 
# replaces the tag name myDevice with device
- rule          : test-logrenameannotation
 action        : logRenameAnnotation
 key           : myDevice
 newkey        : device

logLimitLength

Truncate or drop log tags if tag value length exceeds the limit.

Available action subtypes are truncate, truncateWithEllipsis, and drop.

Parameters
Parameter Description
action loglimitLength
scope Rule applies to the value of the specified log tag (annotation) key.
actionSubtype Allows you to determine how we limit length:
  • drop—Drops requested scope if the value is greater than maxLength. You can't use DROP with the source name.
  • truncate—Truncates requested scope if the value is greater than maxLength.
  • truncateWithEllipsis—Truncates the requested scope if the value is greater than maxLength but preserves the ellipsis (three dots). maxLength must be at least 3 for this action type.
maxLength The maximum length of a log tag value. The length of the input must be greater than the maxLength for rule to be applied.
match (optional) Regular expression. If specified, remove a tag if its value matches this regular expression.
Example 
# Truncate messages longer than 1000 characters
- rule          : test-loglimitlength
  action        : logLimitLength
  maxLength     : 1000
  scope         : message
  actionSubtype : truncate

logBlock

Defines a regex that the log tags must match to be filtered out.

Parameters
Parameter Description
action logBlock
scope Rule applies to the value of the specified log tag key after the value is parsed.</li> </ul>
match A regex pattern that input lines must match to be filtered out.
Examples 
# reject all logs that contain tagToBlockList:[only lower case alphabets]
- rule          : test-logBlock
  action        : logBlock
  match         : "^[a-z]+"
  scope         : tagToBlockList

logAllow

Points must match the allow list to be accepted. Multiple allow rules are allowed. A log tag must match all rules.

Parameters
Parameter Description
action logAllow
scope Rule applies to the value of the specified log tag key after the value is parsed.</li> </ul>
match A regex pattern that input lines must match to be accepted.
Examples 
# only allow logs that contain tagToWhiteList:[only numbers]
 - rule          : test-logAllowRegex
   action        : logAllow
   match         : "^[0-9]+"
   scope         : tagToAllowList
​
# removes all annotations not in the specified list
 - rule: test-logAllowAnnotations
   action: logAllow
   allow:
     - customTag1
     - tagToExtract
     - extractedTag
     - device
     - tagToAllowList

Learn More!

getting started