For best performance and cost reduction, VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront) supports limits. Some limits are recommendations–if your environment exceeds the limits, you’ll see significant performance issues. Other limits result in an error if you exceed the limit.
Concurrent Query Limits
Our service enforces the following concurrent query limits. These limits are subject to change without notice.
Per Customer Concurrent Query Limit
Our service enforces a limit on concurrent queries for each customer cluster. The default is 1000. If you are getting repeated errors that your cluster is exceeding this limit, contact us.
The following error results if your environment exceeds this limit:
HTTP 429 TOO_MANY_REQUESTS
Customer concurrent query limit exceeded. Please try again later. Contact support@wavefront.com for help.
Per User Concurrent Query Limit
Our service enforces a limit on per-user concurrent queries. The default limit is 100. Contact us if you believe that the setting doesn’t make sense for one of your users (for example, one of your service accounts) and we’ll discuss options with you.
The following error results if one of the users exceeds this limit:
HTTP 429 TOO_MANY_REQUESTS
“User concurrent query limit exceeded. Please try again later. Contact support@wavefront.com for help.”
Timeout Limits
| Category | Timeout Limit | Explanation and Best Practice |
|---|---|---|
| Query | 300s (5min) | If a query does not complete in 300s (5 minutes), the chart/API times out. Best practice: Use specific sources and/or point tags in the queries to drill down into specific data that is required. |
| Alert | 60s (1min) | When you create an alert, you define a query to get the data you want to monitor. This query must complete in less than 60 seconds (1 minute). If the query does not get the results in less than 60 seconds, you won't be able to save the alert. Best practice: Use specific sources, point tags, or both in the queries to drill down into specific data that you need. |
| Dynamic Dashboard Variable | 60s (1min) | Dynamic dashboard variables dropdown menus allow users to make a selection that is based on metadata of a query, for example, a set of sources. If the dynamic variable query does not complete in 60s, it is cancelled by the server to avoid a bottleneck in loading a dashboard. Best practice:Do not use wildcard characters in dynamic variables. Use specific sources, point tags, or both in the queries to drill down into specific data that is required. |
| Derived Metric | 300s (5min) | Derived metrics can synthetically create metrics based on existing metrics. The query engine can reingest those metrics. The query that runs for a derived metric, like a regular query, has a 300s or 5 minute timeout. Best practice: Use specific sources and/or point tags in the queries to drill down into specific data that is required. |
Default Customer-Specific Limits
You can start either as part of a free trial or as a new customer. In both cases, a set of out-of-the-box limits applies to that customer account. You can contact our customer success team to request changes. In some cases, a change might involve additional costs.
| Metric length limit | Maximum number of characters for a metric name. | 256 |
| Histogram length limit | Maximum number of characters for a histogram name. | 256 |
| Span length limit | Maximum number of characters for a span name. | 128 |
| Host length limit | Maximum number of characters for a source name. | 128 |
| Annotations count limit | Maximum number of point tags associated with a metric. | 20 |
| Annotations key length limit | Maximum number of characters in a point tag key. | 64 |
| Annotations value length limit | Maximum number of characters in a point tag value. | 255 |
| Counter length limit | Maximum number of characters in a counter metric. | 256 |
| Span annotations count limit | Maximum number of point tags associated with a span. | 50 |
| Span annotations key length limit | Maximum number of characters associated with a span point tag key. | 128 |
| Span annotations value length limit | Maximum number of characters associated with a span point tag value. | 128 |
| Span topology processing Ttl | 10 | |
| Span topology dimensions | Dimensions associated with a span. Defaults to "application" "cluster" "shard" | 128 |
| Span logs size limit | Maximum size of a span log. | 32768 |
| Log annotations count limit | Maximum number of tags associated with a log. | 100 |
| Log annotations key length limit | Maximum number of characters associated with a log tag key. | 128 |
| Log annotations value length limit | Maximum number of characters associated with a log tag value. | 128 |
| Log length limit | Maximum number of character on a log message. | 20000 |
| Search time window for logs | The default search time window for the Log Browser | 15 minutes |
| Metrics to logs tag mapping | Maps the tags of the metrics to the the tags of logs. | source tag |
| Traces to logs tag mapping | Maps the tags of the traces to the the tags of logs. | traceId, source, application, and service tags |
Data Format Best Practices
Follow best practices to avoid hitting query limits and for improved query execution speed and meaningful results.
- Make the metrics the most stable part of your data:
- Do not include source names in the metric name. Our service captures sources separately.
- Do not include data or timestamps in the metric name. Each point has an associated timestamp.
- Aim for a metric hierarchy:
- Partition the top level of the metric hierarchy by including at least one dot.
- Organize metric names in a meaningful hierarchy from most general to most specific (i.e.
system.cpu0.loadavg.1minstead of1m.loadavg.cpu0.system)
- For best performance, keep the number of distinct time series per metric and host to under 1000.
See Operations for Applications Data Naming for more best practices.
More Info
You can examine what’s going on with your cluster in several ways: