Pairing Up Matching Series

Certain operators and functions apply to pairs of time series. When you specify two ts() expressions as parameters, Wavefront implicitly performs series matching across these expressions to identify meaningful pairs of individual time series to operate on. For example, implicit series matching in the following operation causes Wavefront to compare metrics for disk reads and disk writes only when they come from the same source and have common point tag values:

ts(~sample.disk.bytes.read) > ts(~sample.disk.bytes.written)

Implicit series matching also determines whether these operators and functions return any result at all. For example, when you try to subtract one time series from another, Wavefront can’t perform the operation if none of the sources match.

When Wavefront Performs Implicit Series Matching

Wavefront performs series matching when you apply certain operators and functions to two or more unique ts() expressions, each representing two or more unique metric|source|point tag value tuples. The following operators and functions result in series matching:

• Arithmetic operators (+, -, /, *)
• Boolean operators (and, or)
• Comparison operators (>, <, =, >=, <=, !=)
• Filter functions (highpass, lowpass, min, max)
• Exponential and trigonometric functions

In the examples below, the results listed to the right of = represents the set of series that would be displayed.

Series Matching Occurs

The following examples show when series matching occurs:

Series Matching Does Not Occur

The following examples show when series matching does not occur:

Series Matching Basics

Assume you enter the following ts() expression

ts("stats.servers.MemTotal", tag="dc1") - ts("stats.servers.MemFree", tag="east")

Wavefront determines which time series match up and subtracts the value for stats.servers.MemTotal from stats.servers.MemFree for each matching series.

Assume that the source tags dc1 and east have three sources that match up (app-3, app-4, app-5), and four sources that don’t (app-1, app-2, app-6, app-7). As a result, the chart displays only data associated with app-3, app-4, and app-5. Data for app-1, app-2, app-6, and app-7 are discarded.

There are cases when you apply functions to expressions, but no series matching occurs. This happens when one of the evaluated ts() expression is a constant value, such as 1, or represents a single time series, such as a single source or aggregated data with no “group by”.

For example, if you replaced tag="east" with source="app-4", then the value associated with app-4 in the second expression at each time slice is subtracted from each represented source in the first expression at each time slice. If you still want series matching to occur in the previous example, then you can wrap the operator or function with an inner join (i.e. [+]).

Series Matching Example

Here’s a simple example where the Wavefront UI displays a message that informs you that some of the series are not included in all queries.

The reason we get this message is that some expressions limit the environment to env=dev and other expressions don’t use the filter. When part of a query uses a filter, but another part doesn’t, then the whole query uses the filter. In this example, all queries will be limited to env=dev

Series Matching with Point Tags

Consider the following ts() query:

ts(disk.space.total, tag=az-1 and env=*) - ts(disk.space.used, tag=az-1 and env=*)

In this example, the env point tag key takes the values production and development. If source app-1 includes the env value development in the first ts() call, but includes the env value production in the second ts() call, they do not match up.

Series matching occurs only for exact matches. This also means that if two series have the same source|metric|point tag but one of the series includes an additional point tag that the other series does not have, series matching does not include the series with the additional point tag in the results.

Series Matching with the “by” Construct

In some cases, series matching with point tags results in no data because not all of the tags exist on both sides of the operator. You can use the by construct to perform matching using the element of your choice to get results for those series.

Consider the following example:

You’re interested in the set of hosts that have a cpu.idle of more than 50 and a build.version equal to 1000. You start with a set of hosts and run the following query:

(ts(cpu.idle) > 50) and (ts(build.version) = 1000)

The following series are returned by the first part of the query, (cpu.idle) > 50:

The following series are returned by the second part of the query, (build.version) = 1000

It seems like an operation on these two series should yield a result, but the query with the AND operator above returns NO DATA because the dc tag cannot be matched on both sides of the expression.

In this example, while there is a host-1 on both sides of the operation, the first part of the query maps to two different hosts named host-1. There’s no guidance on which of these 2 hosts to pick, so the system doesn’t pick one.

You can use the by query language keyword to specify the point tag(s) to map by. For the example above, you can expand the query as follows:

(ts(cpu.idle) > 50) and by (stage, source) (ts(build.version) = 10000)

With this addition, the query returns the following 6 series, joined with the elements on the right.

Automatic Query Flip

Wavefront automatically flips the query to have the more detailed side of the join be the driver. In the example above, that is the cpu.idle part of the query.