Learn about the query syntax, operators, and functions supported by Wavefront Query Language.

Query Elements

Term Definition
metric The name of a metric. For example: cpu.load.metric
source The name of the entity that emitted the metric. Specify source names with the keyword source. For example:
source=appServer15
source tag A type of source metadata. Specify source tags with the keyword tag. For example:
tag=app.*
point tag A type of custom metric metadata. Point tags have keys and values. For example:
region=us-west-2b
timeWindow A window of time specified in seconds, minutes, hours, days or weeks (1s, 1m, 1h, 1d, 1w), or in terms of the window length you are currently looking at (1vw). If the unit is not specified, the default is minutes.
  • Example. 3h specifies a window of 3 hours.
  • Example. If you are looking at a 30 minute window, 1vw is one view-window length, and therefore equivalent to 30m.
expression An expression consisting of a ts() expression, constant, or combination of ts() expressions and constants. See Expressions.

See Organizing with Tags for information on the different types of tags and how to use them.

Note: Do not use names of functions such as default or sum or other query language elements to name a metric, source, source tag, point tag, or point tag value. If you must, surround the element with double quotes. For example, if you’re using a point tag named default, use "default".

Expressions

An expression may be a ts() expression, a constant, or an arithmetic or Boolean combination of a ts() expressions and constants.

Term Definition
ts() expression Returns all points that match a metric name, filtered by source names, alert names, source tags, alert tags, and point tags.
  • Syntax:
    ts(<metricName>,
      [source=<sourceName>] [and|or]
      [tag=<sourceTagName>] [and|or]
      [<pointTagKey1>=<pointTagValue1>[and|or] ... <pointTagKeyN>=<pointTagValueN>])
    
  • For metric, source, source tag, and point tag naming conventions, see Wavefront Data Format.
  • Sources, source tags, alert names, alert tags, and point tags are optional. For example, to return points from all sources sending the my.metric metric, specify ts(my.metric).
constant A number such as 5.01, 10000, or 40. Constants can be plotted by themselves and composed in expressions using arithmetic operators.
  • You can use SI prefixes(k, M, G, T, P, E, Z, Y) to scale constants by multiples of 1000. G (billion) and T (trillion) are useful when working with network and I/O metrics.
  • Example. Typing 1M is equivalent to typing 1000000
  • Example. Typing 7.2k is equivalent to typing 7200
wildcard Matches strings in metric names, source names, alert names, source tags, alert tags, and point tags.
  • A wildcard is represented with a "*" character. Wavefront supports no other wildcard characters.
  • Example. When filtering sources, match all sources starting with "app-1" (namely, app-10, app-11, app-12, and so on):
    source=app-1*
  • Example. When filtering point tags, match the time series that have <pointTagKey> with any value, and filter out any time series without <pointTagKey>:
    <pointTagKey>="*"
  • Example. When filtering point tags, find any time series that do not have the specified point tag.
    not <pointTagKey>="*"

Operators

All operations between expressions are subject to the matching processes described in Series Matching​.

  • Boolean operators - combine ts() expressions and constants and the filtering performed by source names, alert names, source tags, alert tags, and point tags.
    • and: Returns 1 if both arguments are nonzero. Otherwise, returns 0.
    • or: Returns 1 if at least one argument is nonzero. Otherwise, returns 0.
    • not: Use this operator to exclude a source, tag, or metric. See the examples below.
    • [and], [or]: Perform strict ‘inner join’ versions of the Boolean operators. Strict operators match metric|source|point tag combinations on both sides of the operator and filter out unmatched combinations.
  • Arithmetic operators
    • +, -, *, /: Match metric, source, and point tag combinations on both sides of an expression. If either side of the expression is a ‘singleton’ – that is, a single metric, source, or point tag combination–it automatically matches up with every element on the other side of the expression.
    • [+], [-], [*], [/]: Perform strict ‘inner join’ versions of the arithmetic operators. Strict operators match metric|source|point tag combinations on both sides of the operator and filter out unmatched combinations.
  • Comparison operators
    • <, <=, >, >=, !=, =: Returns 1 if the condition is true. Otherwise returns 0. Double equals (==) is not a supported Wavefront operator.
    • [<], [<=], [>], [>=], [=], [!=]: Perform strict ‘inner join’ versions of the comparison operators. Strict operators match metric|source|point tag combinations on both sides of the operator and filter out unmatched combinations.
  • Examples
    • (ts(my.metric) > 10) and (ts(my.metric) < 20) returns 1 if my.metric is between 10 and 20. Otherwise, returns 0.
    • ts(cpu.load.1m, tag=prod and tag=db) returns cpu.load.1m for all sources tagged with both prod and db.
    • ts(db.query.rate, tag=db and not source=db5.wavefront.com) returns db.query.rate for all sources tagged with db, except for the db5.wavefront.com source.
    • ts("smp-fax*.count" and not "smp-fax*.metrics.wavefront.", source="-eq*" returns all metrics that match "smp-fax*.count" except for those matching "smp-fax*.metrics.wavefront.*".

Tags in Queries

  • Source tags are a way to group sources together. For example, if you have two sources, appServer15 and appServer16, you could add the source tag app to both of them to specify that they are both app servers. Source tags aid in querying by grouping sources together. You can query ts(cpu.load.metric, tag=app) instead of ts(cpu.load.metric, source=appServer15 or source=appServer16). Both queries yield the same result as long as the app tag is added to source=appServer15 and source=appServer16.
  • Alert tags are a way to group alerts together.
  • Point tags are an additional way to describe metrics. An example of a point tag is region=us-west-2b.
  • Example: To query a point cpu.load.metric, source app2, and point tag region=us-west-2b, specify ts(cpu.load.metric, region=us-west-2b and source=app2).

For an overview of tags, see Organizing with Tags.

Variables in Queries

  • A query line variable allows you to refer to a query line as a variable in another query field within the same chart. The query line variable name is the same as the query line name and is referenced in another query field with the syntax ${queryLineName}. For example, if you have a query line named queryLine1 with ts(requests.latency) as the expression, you can enter ${queryLine1} in a another query field to reference ts(requests.latency). The query line being referenced must be a complete expression. If a query line variable and dashboard variable have the same name, the query line variable overrides the dashboard variable.
  • An alias defines any ts() expression as an alias within that single query line using a SQL-style "as" expression. The syntax of an alias is: expression as <aliasName>. If you specify expression as myAlias, you reference the alias as $myAlias. You can use $myAlias multiple times in that query line, and define multiple aliases within a query line. We recommend using alias names that are three letters or longer; specifically, you can't use the SI prefixes (such as k, G, or T) as alias names, and numeric characters are allowed only at the end of the alias name ($test123 is ok, but not $1test or $test4test).
  • A dashboard variable is a variable that can be used within any query line in every chart contained in the dashboard. A dashboard variable can replace any string of text, as opposed to a query line variable and alias which must be a complete expression. If you define dashvar in a dashboard, you refer to ${dashvar} within any query line. You can use aliases, query line variables, and dashboard variables in the same query line; indeed, you can use the same variable name for a dashboard and an alias (though we don't recommend it). See Dashboard Variables.

Aggregation and Raw Aggregation Functions

Aggregation and raw aggregation functions provide a way to combine (aggregate) multiple series into a single series. If there are gaps of data, non-raw aggregation functions first interpolate the points of the underlying set of series (up to one day) if at least 1 known value is available. Then the aggregation function iself is applied to the interpolated series. Raw aggregation functions do not interpolate the underlying series before aggregation. Raw functions aggregate data points by time buckets. For further information, see Standard Versus Raw Aggregation Functions.

Function Definition
sum(<expression> [,metrics|sources|sourceTags|pointTags|<pointTagKey> ]) Returns the sum of all series. If there are gaps in the data in expression, they will first be filled in with interpolation.
rawsum(<expression> [,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the sum of all series. Does not perform interpolation.
avg(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the average of all series. If there are gaps in the data in they will first be filled in with interpolation.
rawavg(<expression> [,metrics|sources|sourceTags|pointTags|< pointTagKey>]) Returns the average of all series. Does not perform interpolation.
min(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the lowest value of all series. If there are gaps in the data in the expression, they will first be filled in with interpolation.
rawmin(<expression>[, metrics|sources| sourceTags|pointTags|<pointTagKey>]) Returns the lowest value of all series. Does not perform interpolation.
max(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the highest value of all series. If there are gaps in the data in expression, they will first be filled in with interpolation.
rawmax(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the highest value of all series. Does not perform interpolation.
count(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the number of series that are reporting. If there are gaps of data in the expression, they are first filled in using interpolation if at least 1 known value is available.
rawcount(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the number of series that are reporting. Does not perform interpolation.
variance(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the variance of all series. If there are gaps of data in the expression, they are first filled in using interpolation if at least 1 known value is available.
rawvariance(<expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the variance of all series. Does not perform interpolation.
percentile(<percentileValue><expression>[,metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the percentileValue value of all series. If there are gaps of data in the expression, they are first filled in using interpolation if at least 1 known value is available. For example, if percentileValue is 99, returns the 99th percentile value of all series.
Set percentileValue to 50 for the mean value of all series.
rawpercentile(<percentileValue>,<expression>[ ,metrics|sources| sourceTags|pointTags|<pointTagKey>]) Returns the percentileValue value of all series. For example, if percentileValue is 99, returns the 99th percentile value of all series.

Grouping and Filtering

When aggregating, you can group or filter the results.

  • When you filter, you restrict the query, for example, only to certain sources. You still get one line for aggregation function. To filter, use the element to filter by inside the parenthesis.

  • You can also group the results of a query and display separate lines for the different group members. For example, when grouping by source, you get one line for each source. When grouping by source tags, you get one line for each source tag that is explicitly specified in the ts() expression.

    To group an aggregation by metrics, sources, source tags, all point tags keys, or a specific point tag key, include the
    [, metrics|sources|sourceTags|pointTags|<pointTagKey>] keyword after the ts() expression, separated by a comma.

Filter Example

sum(ts(~sample.cpu.loadavg.1m, source=app-1*)) shows the sum of the values reported for the metric, but only from the sources that match app-1*.

Grouping Examples

  • Group by metrics: sum(ts(cpu.loadavg.1m),metrics)
  • Group by sources: sum(ts(cpu.loadavg.1m),sources)
  • Group by source tags: sum(ts(cpu.loadavg.1m, tag=prod or tag=db),sourceTags)
  • Group by all available point tag keys: sum(ts(cpu.loadavg.1m),pointTags)
  • Group by the region point tag key: sum(ts(cpu.loadavg.1m),region)

Filtering and Comparison Functions

Function Definition
highpass(<expression1>, <expression2>[, inner]) Returns only the points in expression2 that are above expression1. expression1 can be a constant.
lowpass(<expression1>, <expression2>[, inner]) Returns only the points in expression2 that are below expression1. expression1 can be a constant.
min(<expression1>, <expression2>) Returns the lower of the two values in expression1 and expression2. For example: min(160, ts(my.metric)) returns 160 if my.metric is > 160. If my.metric is < 160, returns the value of my.metric.
max(<expression1>, <expression2>) Returns the higher of the two values in expression1 and expression2. For example: max(160, ts(my.metric)) returns 160 if my.metric is < 160. If my.metric is > 160, returns the value of my.metric.
between(<expression>, <lower>, <upper>) Returns 1 if expression is >= lower and <= upper. Otherwise, returns 0. This function outputs continuous time series.
downsample(<timeWindow>, <expression>) Returns the values in expression that occur in each time window. For example: downsample(30m, ts(my.metric)) returns the values of my.metric every half hour.
align(<timeWindow>,[mean|median|min|max|first|last|sum|count,] <expression>) Returns one value in expression for each time window. For example, if you are collecting data once a minute, but you want data points to be displayed every 30 minutes (summarized by median every 30 minutes), use align(30m, median, ts(my.metric)). See Bucketing with align().
topk(<numberOfTimeSeries>, <expression>) Returns the top numberOfTimeSeries series in expression, based on the most recent data point.
bottomk(<numberOfTimeSeries>, <expression>) Returns the bottom numberOfTimeSeries series in expression, based on the most recent data point.
top(<numberOfTimeSeries>, <expression>) Displays the top numberOfTimeSeries series in expression as 1, based on the most recent data point. Displays all other series as 0. This function outputs continuous time series.
bottom(<numberOfTimeSeries>, <expression>) Displays the bottom numberOfTimeSeries series in expression as 1, based on the most recent data point. Displays all other series as 0. This function outputs continuous time series.
filter(<expression> [, <metric>|source=|tagk=]) Retains only the time series in expression that match the specified metric, source, or point tag. No key is required to filter a metric. filter is similar to retainSeries(), but does not support matching a source tag.
retainSeries(<expression> [, <metric>|source=|tag=|tagk=]) Retains only the time series in expression that match the specified metric, source, source tag, or point tag. No key is required to retain a metric.
removeSeries(<expression> [, <metric>|source=|tag=|tagk=]) Suppresses any time series in expression that matches the specified metric, source, source tag, or point tag. No key is required to remove a metric.
sample(<numberOfTimeSeries>, <expression>) Displays a non-random sample set of numberOfTimeSeries time series based on expression. Repeated calls will display the same sample set as long as the underlying set of time series stays the same.
random(<numberOfTimeSeries>, <expression>) Displays a random set of numberOfTimeSeries time series based on expression. Repeated calls always display different sample sets.
limit(<numberOfTimeSeries>[, <offsetNumber>], <expression>) Displays numberOfTimeSeries time series. Use the optional offsetNumber to specify an index to start with.
hideBefore(<timeWindow>, <expression>) Hides data before a specified time. For example, hideBefore(10m) hides data that’s older than 10 minutes.
hideAfter(<timeWindow>, <expression>) Hides data after a specified time. For example, hideAfter(10m) hides data that’s newer than 10 minutes ago.

Standard Time Functions

Function Definition
rate(expression) Returns the per-second change of expression; should be used on monotonic counter metrics (metrics that have values that only increase). Automatically handles zero-resets in counters.
deriv(expression) Returns the per-second change of expression. Does not handle zero-resets, but can be used on non-counter (decreasing) metrics.
lag(timeWindow, expression) Returns an expression time shifted by timeWindow, to enable comparison of an expression with its own past behavior. Starting with release 2018.10, you can shift by timeWindow into the future. Example: lag(3h, ts(my.metric)) returns each point from expression from 3 hours ago.
lead(timeWindow, expression) Returns an expression time shifted by timeWindow into the past to enable comparison of an expression with its own future behavior. For example lead(3h, ts(my.metric)) returns each point from expression 3 hours earlier than its actual timestamp.
at(timeWindow, expression) Returns the value of the expression from timeWindow ago. at() only looks at a single point, and returns a flat line, across all time points. If you query at(2h, ts(my.metric)), you'll see the value of ts(my.metric) from 2 hours ago, even if you're looking at a chart from weeks or months ago. To associate at() with a time at the beginning or end of your current chart window, replace timeWindow with start, end, or now. This function outputs continuous time series.
year("timezone") Returns the 4-digit year for a timezone. Sample timezones include "US/Pacific" and "Europe/London". The list of valid timezones is available at http://joda-time.sourceforge.net/timezones.html. This function outputs continuous time series.
month("timezone") Returns the numerical month for a timezone. This function outputs continuous time series.
dayOfYear("timezone") Returns the day (within the year) for a timezone. Always returns a value from 1 to 366. This function outputs continuous time series.
day("timezone") Returns the day (within the month) for a timezone. Always returns a whole number from 1 to 31. This function outputs continuous time series.
weekday("timezone") Returns the day (within the week) for a timezone. Always returns a whole number from 1 (Monday) to 5 (Friday) for weekdays, and 6 (Saturday) and 7 (Sunday) for weekends. This function outputs continuous time series.
hour("timezone") Returns the hour (within the day) for a timezone. Always returns a decimal value from 0.0 to 24.0. This function outputs continuous time series.
timestamp(expression) Returns the timestamp associated with the reported data point of expression. To see the entire raw value of the timestamp in the legend (in epoch seconds), hold down the shift key when you hover over a point.
time() Returns epoch seconds for each point in time. This function outputs continuous time series.

Moving Window Time Functions

Moving window time functions allow you to calculate continuous aggregation over sliding windows. For further information, see Using Moving and Tumbling Windows to Highlight Trends.

These functions output continuous time series, with the exception of integral().

Function Definition
mavg(timeWindow, expression) Returns the moving average of each series over timeWindow. Example: mavg(60m, ts(my.metric)) returns, at each point, the moving average over the last 60 minutes for each series in expression.
msum(timeWindow, expression) Returns the moving sum of each series over timeWindow. Don't confuse this function with mcount(), which returns the number of data points. Example: msum(10m, ts(my.metric)) returns, at each point, the sum of all the points over the last 10 minutes for each series in expression.
mmedian(timeWindow, expression) Returns the moving median of each series over timeWindow.
mvar(timeWindow, expression) Returns the moving variance of each series over timeWindow. Example: To get the moving standard deviation, apply the sqrt function to mvar: sqrt(mvar(120m, ts(my.metric))).
mcount(timeWindow, expression) Returns the number of data points over timeWindow. If expression stops reporting data, mcount() continues to return data up to 2x the duration of timeWindow before returning no data. See Account for Missing Data Points. Don't confuse this with msum(), which returns the sum of the data points.
mmin(timeWindow, expression) Returns the minimum of each series over timeWindow.
mmax(timeWindow, expression) Returns the maximum of each series over timeWindow.
mpercentile(timeWindow, percentileValue, expression) Returns the percentile of each series over timeWindow. percentileValue must be >= 0 and <= 100.
mseriescount(timeWindow, expression,[metrics|sources|sourceTags|pointTags|<pointTagKey>]) Returns the aggregated number of series reporting over timeWindow. Example: mseriescount(60m, ts(my.metric)) returns, at each point, the number of series reporting over the last 60 minutes
mdiff(timeWindow, expression) Returns the difference between expression and expression's value at timeWindow ago. This is a raw function, so it does not interpolate the points before doing the subtraction. An example would be expression - expression's value 5 minutes ago.
mcorr(timeWindow, expression1, expression2[, inner]) Returns the moving correlation between two time series expressions over timeWindow.
integrate(timeWindow, expression) Returns the moving integration on a given time series expression over timeWindow.
integral(expression) Returns the moving sum over time for the given time series expression over the time interval of the current chart window. Always starts at 0 on the left side of the chart showing the total accumulation over the time duration of the current chart window.
flapping(timeWindow, expression) Returns the number of times a counter has reset within timeWindow.
any(timeWindow, expression) Returns 1 if the expression over timeWindow has been non-zero at any time. Otherwise, it returns 0.
all(timeWindow, expression) Returns 1 if the expression over timeWindow has been non-zero at every time in that window. Otherwise, it returns 0.

Conditional Functions

Function Definition
if(expression, ThenExpression, ElseExpression) Returns ThenExpression if expression >0. Otherwise, returns ElseExpression. Expects a time series expression as a first argument, and, since time series are numeric, only numeric comparisons are supported. When both ThenExpression and ElseExpression return data, if() performs series matching against expression.

Example: If expression is ts(my.metric) >= 10
, if (expression, ts(my.metric), ts(another.metric)) returns ts(my.metric) only when ts(my.metric) >= 10; when ts(my.metric) < 10, it returns ts(another.metric).

When expression and at least one of ThenExpression or ElseExpression is not a constant time series, this function outputs continuous time series.

Rounding Functions

Function Definition
round(expression) Returns the nearest whole number to expression.
ceil(expression) Rounds up expression to the next largest whole number.
floor(expression) Rounds down expression to the next smallest whole number.

Missing Data Functions

Missing data functions allow you to interpolate missing data with points based on other points in a series.

Function Definition
default([,<timeWindow>, ]<delayTime> <defaultValue>, <expression>) Fills in gaps in expression with defaultValue (whether that's a constant or an expression). The optional timeWindow parameter fills in the specified period of time after each existing point (for example, 5m for 5 minutes). Without this argument, all gaps are filled in. The optional delayTime parameter specifies the amount of time that must pass without a reported value in order for the default value to be applied.
last([,<timeWindow>, ] <expression>) Fills in gaps in expression with the last known value of expression. Use the optional timeWindow parameter to fill in a specified time period after each existing point.
next([,<timeWindow>, ] <expression>) Fills in gaps in expression with the next known value of expression. Use the optional timeWindow parameter to fill in a specified time period before the first data point after the missing data.
interpolate(<expression>) Fills in gaps in expression with a continuous linear interpolation of points.

Metadata Functions

Metadata functions help users rename a metric, source, or create a synthetic point tag on a metric. There are three ways to formulate the alias:

  • Node index - Extract a string component based on a zeroBasedNodeIndex. Components are identified by the default delimiter “.” or a delimiter specified in delimiterDefinition.
  • Regular expression replacement - Identify the string using a regular expression and replacement string using a replacement pattern.
  • String substitution - Replace a metric or source in an expression with a replacement string.
Function Definition
aliasMetric(<expression>[,metric|source|{tagk,<pointTagKey>},][zeroBasedNodeIndex[ delimiterDefinition] | "<regexSearchPattern>", "<replacementPattern>" | "<replacementString>")] Returns expression with the metrics renamed with a string extracted from a metric, source, or point tag value of expression. If you don’t specify the metric|source|{tagk, <pointTagKey>} parameter, it defaults to source.
aliasSource(expression[,metric|source|{tagk,<pointTagKey>},] [zeroBasedNodeIndex[ delimiterDefinition] | "regexSearchPattern", "replacementPattern" | "replacementString")] Returns expression with the sources renamed with a string extracted from a metric, source, or point tag value of expression. If you don’t specify metric|source|{tagk, <pointTagKey>}, the parameter defaults to source.
taggify(expression,metric|source|{tagk,<pointTagKey>},<newPointTagKey>, [zeroBasedNodeIndex[ delimiterDefinition] | "regexSearchPattern", "replacementPattern" | "replacementString")] Returns expression with the source renamed with a string extracted from a metric, source, or point tag value of expression. If you don’t specify metric|source|sourceTags {tagk, <pointTagKey>}, the parameter defaults to source.

Examples

  • Node index: aliasMetric(ts(cpu.loadavg.1m, source), 1) the extracted string is selected by node index. The metric cpu.loadavg.1m has 3 components. Setting zeroBasedNodeIndex to 1 extracts the second component (loadavg).
  • Node index with delimiter: cpu-loadavg-1m sets delimiterDefinition to -.
  • String substitution:
    • Original: max(ts(customer.alerts.active), metrics)
    • Renamed: aliasMetric(${original}, "Total Number Of Alerts"), replaces the metric customer.alerts.active with "Total Number Of Alerts".

Exponential and Trigonometric Functions

Function Definition
sqrt(expression) Returns the square root of expression.
pow(expression, expression[, inner]) Raises expression to the power of expression. Wavefront does not support imaginary numbers, so pow(-1, 0.5) returns no data.
exp(expression) Returns the exponential of expression.
log(expression) Returns the natural log of expression.
log10(expression) Returns the log base 10 of expression.
toDegrees(numRadians), toRadians(numDegrees) Convert radians to degrees, and vice versa.
sin(expression), cos(expression), tan(expression),
asin(expression), acos(expression),
atan(expression), atan2(expression),
sinh(expression), cosh(expression), tanh(expression)
Performs the specified trigonometric function on expression interpreted in radians.

Predictive Functions

Function Definition
hw(timeWindow1, timeWindow2, timeWindow3, expression)[, value1, value2, value3]) Returns a smoothed version of expression and forecasts its future points using the Holt-Winters triple exponential smoothing algorithm for seasonal data. See Holt-Winters Predictive Analysis.
  • timeWindow1 is the amount of data we use to smooth the series and to forecast.
  • timeWindow2 is the seasonal length of the data.
  • timeWindow3 is the rate at which the expression should be sampled.
  • The optional three values are coefficients for the Holt-Winters equations, and must be decimals between 0 and 1. If no values are given, Wavefront selects them manually.

Event Functions

Event functions are used to display events in charts and perform conversions on events sets. For further information, see Basic events() Queries and Advanced events() Queries.

Function Definition
events(filters) Returns the set of events that match filters. The available filters are Event Filters. The returned set of events can be passed as an argument to functions that accept events. When passed to a chart query, displays the events. The chart must contain at least 1 ts() expression for events to display.
count(events) Converts events into a single time series, where every data point represents the number of events that started at that time minus the number of events that ended at that time. Instantaneous events are represented as a single "0" value: 1 started minus 1 ended (instantaneous events are defined as events having their end time equal to their start time).
ongoing(events) Returns a continuous time series representing the number of ongoing events at any given moment within the query time window. See [When Does an Event Query Return Events?](events_queries.html#when-does-an-event-query-return-events) for some background information.
closed(events) Returns events that have ended and instantaneous events that occurred in the past.
until(events) Returns synthetic events that start at the beginning of time (Jan 1, 1970) and end where the input events start.
after(events) Returns synthetic ongoing events that start the moment the input events end.
since(events) Returns synthetic events with the same start time and no end time (converts all input events to ongoing).
since(timeWindow) Creates a single synthetic event that started timeWindow ago and ended "now". timeWindow can be specified in seconds, minutes, hours, days or weeks (e.g., 1s, 1m, 1h, 1d, 1w). If the unit is not specified, the default is minutes.
timespan(startTimestamp, endTimestamp) Creates a single synthetic event with the specified start and end timestamps. A timestamp can be expressed in epoch seconds or using a time expression such as "5 minutes ago". Example: timespan("5 minutes ago", "2 minutes ago").
first(events) Returns a single event with the earliest start time.
last(events) Returns a single event with the latest start time.
firstEnding(events) Returns a single event with the earliest end time.
lastEnding(events) Returns a single event with the latest end time.

Example

events(type=alert, name="disk space is low", alertTag=MicroService.App1.*)

Event Filters

Event filters allow you to limit which events are returned from events() queries.

FilterDescriptionExample
alertId The ID of the alert that created the event. events(alertId=1411189741192)
alertTag A tag associated with the alert that generated the event. events(alertTag="ops")
eventTag A tag associated with the event. events(eventTag="codepushes")
name The name of the event. Manually created events have a unique name, while events generated by an alert have the same name as the associated alert. The name filter requires quotes if spaces exist in the name. events(name="Request Latency too high")
severity The classification of the user event or the severity of alert that generated the event. User event classification levels are severe, warn, info, and unclassified. Although an event can be left as unclassified, the severity filter does not accept unclassified as a valid value. events(severity="info")
source or tag The source or source tag associated with the alert that generated the event. The source filter allows you to display events generated by an alert based on a single source or set of sources. The tag filter works the same way, but allows you to specify a source tag instead of a source name. events(source="app-*" or tag="dc2")
subtype The subtype of event of type alert-detail: failing, recovered. events(subtype="failing")
target The target of the alert that generated the event. The list of targets associated with an alert are considered a single string. If you want to identify a single target within that string, then you must use wildcards. For example, if the notification field contains: john.doe@example.com, jane.doe@example.com, pd:fbw21c9ee0219473w2179r4t23f8c34, to use jane.doe@example.com as the target filter, specify the email as *jane.doe@example.com*. events(target="*jane.doe@example.com*")
type The type of an event. There are system-generated event types: alert, maintenanceWindow, alert-detail, credentials-error, alert-created, alert-updated, alert-deleted, dashboard-deleted, maintenancewindow-created, maintenancewindow-updated, maintenancewindow-deleted and you can optionally assign a type to a user event. The value requires quotes if it contains spaces or starts with a wildcard. events(type="Code push")
events(type="*-created")

Miscellaneous Functions

Function Definition
collect(expression, expression2, expression3, ...) Returns a ts() expression that is the combination of two or more ts() expressions. The returned expression includes a synthetic collect_<number> point tag, where <number> is the number of input expressions.
exists(expression) Returns 1 if at least one value in expression has been reported in the last 4 weeks. Otherwise, it returns 0. This function outputs continuous time series.
abs(expression) Returns the absolute value of expression.
random() Returns random values between 0.0 and 1.0. If you reload a chart that uses random(), the reloaded chart returns new random values. This function outputs continuous time series.
normalize(expression) Returns every series in expression scaled so it has a minimum of 0 and a maximum of 1.0.
haversine(expression1, expression2, expression3, ...) Returns the distance between coordinates. expression(s) can be constants or ts() expressions.

Troubleshooting

ProblemCauseResolution/Details
A time series you send to Wavefront is discrete, for example, you send data points every minute, but the data appear as continuous. Continuous means that you see data every second (or time slice) regardless of the interval of the underlying data. Some functions return continuous time series even if the underlying metrics are discrete. See the list on the right. The following functions return continuous time series.
  • Moving time windows except integral.
  • Missing data functions.
  • The if() function when expression is not a constant time series
  • The ongoing(), exists(), and random() functions.
  • The at(), year(), month(), dayOfYear(), day(), weekday(), hour(), and time() functions.
  • The between(), top(), and bottom() functions.
  • Constant time series functions: at(), top(), bottom(), <number>
After entering a query expression the following error displays: Query syntax error: Missing expression argument. An expression argument to a function is not well-formed. Build up the expression by small steps ensuring that the expression is valid at each step.
You see the warning indicator in a chart and a warning like the following:

The expression: ts(<metric>, source=<source>) has been pre-aligned, making it equivalent to align(120s, mean, ts(<metric>, source=<source>)) in order to improve the performance of the sum() aggregation operation. You can wrap the expression with align() to explicitly state the periodicity and desired summarization strategy.

Assuming an original query of sum(ts(<metric>, source=<source>)), Wavefront has pre-aligned the series to improve performance. Depends on the use case. For details, see The align() Function.