The Wavefront Query Language lets you retrieve and display the data that has been ingested into Tanzu Observability by Wavefront and create alerts that use this data.
Time series data | The query language is particularly well suited to manipulating time series data because it accommodates the periodicity, potential irregularity, and streaming nature of that type of data. |
Histograms | The query language includes functions for manipulating histograms. |
Traces and spans | Use the tracing UI to query traces and spans. |
Video: Optimize Dashboard Performance
Watch this video to learn how to optimize dashboard and query performance.
Use Performance Statistics
You can see performance statistics for the whole chart and for each query of the chart. For the performance statistics, we measure the following characteristics:
- Cardinality: Number of unique time series. A unique time series has unique metric name, source name and point tags (key and value). For example, you might receive
networks_bytes_received
from multiple sources and with multiple point tags (e.g.availability_zone
). You can lower cardinality for each query (and the chart) by filtering, for example, limiting the query to certain sources, certain availability zones, etc. - Points Scanned: Number of data points that were queried to show the chart on the screen. You can affect this number by including the time window in the query or by changing the time window interactively.
- Duration: Time between query start and return of result.
View Chart Statistics
To see the overall performance statistics for a chart:
The chart stats window opens. You can move the chart stats window within the chart borders. |
![]() |
View Query Statistics
To see the performance statistics for a particular query of a chart or alert:
|
![]() |
Use Performance Improvement Suggestions
If the query uses certain functions in ways that often cause performance degradation, Tanzu Observability shows actionable suggestions for improving the query performance. The suggestions also include links to documentation and videos for details.
A dot symbol on the lightbulb icon for a query indicates that Tanzu Observability has suggestions for improving the query performance. | ![]() |
To see and, optionally, apply the performance improvement suggestions for a query:
|
![]() |
Use Filters to Look at the Right Data
For best query language performance, it’s important to look at just the right amount of data.
Use filtering functions in queries narrow down the query space. For example, if a query filters metrics by source or point tag, the query returns faster because the query engine knows which metrics to fetch. Here are some tips:
-
Filter by source: By default, if you query a metric such as
cpu.loadavg.1m
, the query engine retrieves that metric for any source (host, container, etc.). To significantly improve query performance, query only for sources that you need to know about.Example:
- Faster:
ts(~cpu.loadavg.1m, source="db-1")
narrows down the query to a specific time series. - Slower:
ts(~cpu.loadavg.1m)
returns all time series and is slower.
- Faster:
-
Filter by point tag: If your data comes in with point tags, such as the availability zone, environment, or other attribute, you can change your query to filter by point tag.
Example:
ts(~cpu.loadavg.1m AND source=app-* AND env="production")
returns only metrics with sources that start withapp-
and that also have the valueproduction
for theenv
point tag.
-
Avoid NOT in filters: With
AND NOT
, the query engine has to search through everything matching the metric, and then filter.Example:
- Faster:
ts(~cpu.loadavg.1m, source="db-1" and env="prod")
narrows down the query to a specific time series. - Slower:
ts(~cpu.loadavg.1m AND NOT env="dev")
is more expensive. WithAND NOT
the query engine has to search through all instances of~cpu.loadavg.1m
and extract instances that do not have theenv-"dev"
point tag.
- Faster:
-
Filter in the base query: If possible filter in the base query instead of using advanced filtering functions.
Example:
- Faster:
sum(ts(user.metric, source=app-1)))
- Slower:
retainSeries(sum(ts(user.metric)), source=app-1))
- Faster:
Be Smart About Aggregation
Aggregation functions like sum() or avg() let you combine different time series, for example, by showing the sum or average of a set of time series. For optimal accuracy, the query engine uses interpolation. After interpolation, each time series has a value at each point in time which improves accuracy during aggregation, but affects performance. See Aggregating Time Series for background and a video.
You have these options to eliminate the overhead from interpolation:
Use align() with Aggregation Functions
The align() function changes how bucketing happens.
Example:
- More precise:
avg(ts(~sample.network.bytes.sent))
returns the average over all time series, inserting points so there’s a value for each time series at any time there’s a value for one time series. - Faster:
align(1m, mean, ts("my.metric"))
returns the average over all time series, and uses the values at each 1 minute point in time.
In certain cases, the query engine performs prealignment.
Use Raw Aggregation Functions
Instead of using align(), you can avoid the overhead of interpolation with a raw aggregation function. Aggregating Time Series has details and a video.
- Standard aggregation functions (e.g. sum(), avg(), or max()) first interpolate the points of the underlying set of series, and then apply the aggregation function to the interpolated series. These functions aggregate multiple series down, usually to a single series.
- Raw aggregation functions (e.g. rawsum(), rawavg()) do not interpolate the underlying series before aggregation.
Example:
- More precise:
sum(ts(~sample.cpu.loadavg.1m, source=app-1*))
performs interpolation first, and then computes the sum. - Faster:
rawsum(ts(~sample.cpu.loadavg.1m, source=app-1*))
does not perform interpolation and computes the sum from the raw data.
Specify a Time Window with Missing Data Functions
Missing data functions such as last() support an optional timeWindow
parameter. The default() function also supports a delayTime
parameter. If you don’t specify those time parameters, the query engine applies the default value for every second and for gaps up to 28 days. This impacts performance of the query and the dashboard.
- Faster:
default([<timeWindow>,] [<delayTime>,] <defaultValue>, <tsExpression>)
- Slower:
default(0, <tsExpression>)
The time window is a measure of time expressed as an integer number of units. The default unit is minutes. You can specify:
- Seconds, minutes, hours, days, or weeks (1s, 1m, 1h, 1d, 1w). For example, 3h specifies 3 hours.
- Time relative to the window length of the chart you are currently looking at (1vw). If you are looking at a 30-minute window, 1vw is one view-window length, and therefore equivalent to 30m.
- Time relative to the bucket size of the chart (1bw). The query engine calculates bucket size based on the view window length and screen resolution. You can see bucket size at the bottom left of each chart.
Use Wildcard Characters with Care
WQL supports the asterisk (*) as a wildcard character. Wildcards in queries can result in many time series on a chart, which can be confusing and affect performance. If using a wildcard character make sense for your use case, use delimiters, and don’t use a wildcard at the beginning of a query.
- Faster:
ts(‘abc.*.xyz’)
– Using delimiters around wildcards. - Slower:
ts(“abc*xyz”)
– Not using a period as a delimiter. - Slower:
ts("*abc.xyz")
– Wildcard character at the beginning of a query.