This document describes Wavefront proxy 4.12 configuration options. For changes since previous proxy versions, see Wavefront Proxy Versions.
Installing a Proxy
To install a proxy, follow the directions in Installing Wavefront Proxies. The installation procedures perform basic configuration. For advanced configuration, see the options in the next section.
Proxy Configuration
Paths
In this section, file paths use the following conventions and values:
<wavefront_config_path>- Linux -
/etc/wavefront/wavefront-proxy - Mac -
/usr/local/etc/wavefront/wavefront-proxy - Windows -
C:\Program Files (x86)\Wavefront\conf
- Linux -
<wavefront_log_path>- Linux -
/var/log/wavefront - Mac -
/usr/local/var/log/wavefront - Windows -
C:\Program Files (x86)\Wavefront
- Linux -
<wavefront_spool_path>- Linux -
/var/spool/wavefront-proxy - Mac -
/usr/local/var/spool/wavefront-proxy - Windows -
C:\Program Files (x86)\Wavefront\bin
- Linux -
Configuration Properties
The main Wavefront proxy configuration file is maintained in <wavefront_config_path>/wavefront.conf. Besides the server and hostname properties, the configuration file offers many options for changing how the proxy processes your data. There are optional configuration files for rewriting metrics and parsing log data. None of these need to be changed from their default values, but can be adjusted for your particular needs. After changing a configuration option, restart the proxy service.
| Property | Purpose | Format | Example | Since Version |
|---|---|---|---|---|
| agentMetricsPointTags | Point tags and their values to be passed along with ~agent.* metrics. Default: None. |
A comma-separated list of key-value pairs. | dc=west,env=prod | 3.24 |
| blacklistRegex | Regex pattern (java.util.regex) that input lines must match to be filtered out. Input lines are checked against the pattern as they come in and before the prefix is prepended. | A valid regex pattern. | Filter out points that begin with qa., development., or test.: ^(qa|development|test). |
3.1 |
| buffer | Location of buffer files for saving failed transmissions for retry. | A valid path on the local file system. | <wavefront_spool_path>/buffer |
3.20 |
| customSourceTags | Point tag keys to use as 'source' if no 'source' or 'host' field is present. Default: fqdn, hostname. | A comma-separated list of point tag keys. | fqdn, hostname | 3.14 |
| dataBackfillCutoffHours | The cut-off point for what is considered a valid timestamp for back-dated points. We do not recommend setting this value larger than 1 year unless backfilling or migrating historic data. Default: 8760 (1 year), so all points older than 1 year are rejected. | A positive integer. | 8760 | 4.1 |
| ephemeral | Whether to automatically clean up old and orphaned proxy instances from the Wavefront Agents page. We recommend enabling ephemeral mode if you're running the proxy in a container that may be frequently spun down and recreated. Default: false. | Boolean | true | 3.14 |
| fileBeatPort | TCP port to listen on for Filebeat data. Default: 5044. | A port number. | 5044 | 4.1 |
| flushThreads | Number of threads that flush data to the server. Setting this value too high will result in sending batches that are too small to the Wavefront server and wasting connections. Values between 6 and 16 are a good starting point. This setting is per listening port. Default: The number of available processors (min 4). | A positive integer. | 16. | 3.14 |
| graphitePorts | TCP ports to listen on for Graphite data. Define which of the segments in your Graphite metrics map to a hostname in the graphiteFormat property. Default: 2003. | A comma-separated list of available port numbers. Can be a single port. | 2003 2003,2004 |
|
| graphiteFormat | Indexes of fields within Graphite and collectd metric names that correspond to a hostname. For example, if your metrics have the format: collectd.prod.www04.cpu.loadavg.1m specify the 3rd and 2nd indexes (www04.prod) to be extracted and treated as the hostname. The remainder collectd.cpu.loadavg.1m is treated as the metric name. |
A comma-separated list of indexes. | 3,2 4,2,5 3 |
|
| graphiteDelimiters | Characters that should be replaced by dots, in case they were escaped within Graphite and collectd before sending. A common delimiter is the underscore character; so if you extract a hostname field with the value web04_www, it is changed to web04.www. |
A concatenation of delimiter characters, without any separators. | - | |
| hostname | A name unique across your account representing the machine that the proxy is running on. The hostname is not used to tag your metrics; rather, it's used to tag proxy metrics, such as JVM statistics, per-proxy point rates, and so on. | A string containing alphanumeric characters and periods. | ||
| httpConnectTimeout | HTTP connect timeout (in milliseconds). Default: 5000 (5s). | A positive integer. | 5000 | 4.1 |
| httpRequestTimeout | HTTP request timeout (in milliseconds). We do not recommend setting this value to be higher than 20000. Recommended value for most configurations is 10000 (10 seconds). Default: 20000 (20s). | A positive integer. | 10000 | 4.1 |
| httpUserAgent | Override User-Agent in request headers. Can help bypass excessively restrictive filters on the HTTP proxy. Default user agent: Wavefront-Proxy/<version>. | A string. | 'Mozilla/5.0' | 4.1 |
| idFile | Location of the PID file for the wavefront-proxy process. Default: <wavefront_config_path>/.wavefront_id. |
A valid path on the local file system. | ||
| jsonListenerPorts | TCP ports to listen on for incoming JSON-formatted metrics. Default: None. | A comma-separated list of available port numbers. Can be a single port. | ||
| logsIngestionConfigFile | The file containing instructions for parsing log data into metrics. See Log Data Metrics Integration.
Default: <wavefront_config_path>/logsIngestion.yaml. |
A valid path on the local file system. | 4.1 | |
| opentsdbPorts | TCP ports to listen on for incoming OpenTSDB-formatted data. Default: None. Default: 4242. | A comma-separated list of available port numbers. Can be a single port. | 4242 | 3.1 |
| picklePorts | TCP ports to listen on for incoming data in Graphite pickle format (from carbon-relay). Default: None. | A comma-separated list of available port numbers. Can be a single port. | 5878 | 3.20 |
| prefix | String to prepend before every metric name. For example, if you set prefix to ‘production’, a metric that is sent to the proxy as cpu.loadavg.1m is sent from the proxy to Wavefront as production.cpu.loadavg.1m. You can include longer prefixes such as production.nyc.dc1, and so on. Default: None. |
A lowercase alphanumeric string, with periods separating segments. You do not need to include a trailing period. | production production.nyc.dc1 |
|
| preprocessorConfigFile | Path to the optional preprocessor config file containing preprocessor rules for filtering and rewriting metrics. Default: None. | A valid path on the local file system. | <wavefront_config_path>/preprocessor_rules.yaml |
4.1 |
| proxyHost | HTTP proxy host to be used in configurations when direct HTTP connections to Wavefront servers are not possible (must be used with proxyPort). | A string. | proxy.local | 3.23 |
| proxyPassword | When used with proxyUser, sets credentials to use with the HTTP proxy if it requires authentication. | A string. | validPassword123 | 3.23 |
| proxyPort | HTTP proxy port to be used in configurations when direct HTTP connections to Wavefront servers are not possible (must be used with proxyHost). | A port number. | 8080 | 3.23 |
| proxyUser | When used with proxyPassword, sets credentials to use with the HTTP proxy if it requires authentication. | A string. | validUser | 3.23 |
| pushBlockedSamples | Number of blocked points to print to the log immediately following each summary line (every 10 flushes). If 0, print None of them. If you are seeing a non-zero number of blocked points in the summary lines and want to debug what that data is, we recommend setting this to 5 or so. Default: 0. | 0 or a positive integer. | 5 | |
| pushFlushInterval | Milliseconds to wait between each flush to Wavefront. Default: 1000. | An integer equal to or greater than 1000. | 1000 | |
| pushFlushMaxPoints | Maximum number of points to send to Wavefront during each flush. Default: 40,000. | A positive integer. | 40000 | |
| pushListenerPorts | TCP ports to listen on for incoming data. Default: 2878. | A comma-separated list of available port numbers. Can be a single port. | 2878 2878,2879,2880 |
|
| pushLogLevel | Frequency to print status information on the data flow to the log. SUMMARY prints a line every 60 flushes, while DETAILED prints a line on each flush. | None, SUMMARY, or DETAILED | SUMMARY | |
| pushMemoryBufferLimit | Maximum number of points that can stay in memory buffers before spooling to disk. Setting this value lower than default reduces memory usage but forces the proxy to queue points by spooling to disk more frequently, if you have points arriving at the proxy in short bursts. Default: 16 * pushFlushMaxPoints. Minimum: pushFlushMaxPoints. | A positive integer. | 640000 | 4.1 |
| pushRateLimit | Maximum number of points to send to Wavefront per minute. Default: unlimited. | A positive integer. | 20000 | 4.1 |
| pushValidationLevel | Level of validation on incoming data that should be performed before sending the data to Wavefront. If NO_VALIDATION, all data is sent forward. If NUMERIC_ONLY, data is checked to make sure that it is numerical and dropped locally if it is not. | NUMERIC_ONLY or NO_VALIDATION | NUMERIC_ONLY | |
| rawLogsPort | TCP port to listen on for log data. Default: 5045. | A port number. | 5045 | 4.4 |
| retryBackoffBaseSeconds | For exponential back-off when retry threads are throttled, the base (a in a^b) in seconds. Default: 2.0. | A positive number, integer or decimal. | 2.0 | |
| retryThreads | Number of threads retrying failed transmissions. If no value is specified, it defaults to the number of processor cores available to the host or 4, whichever is greater. Every retry thread uses a separate buffer file (capped at 2GB) to persist queued data points, so the number of threads effectively controls the maximum amount of space that the proxy can potentially use to buffer points locally. | A positive integer. | 4 | |
| server | The API URL of the Wavefront server in the format https://<wavefront_instance>.wavefront.com/api/. |
|||
| soLingerTime | Enable SO_LINGER with the specified linger time in seconds. We recommend setting this value to 0 when running in a high-availability configuration under a load balancer.Default: 0 (disabled). | 0 or a positive integer. | 0 | 4.1 |
| splitPushWhenRateLimited | Whether to split the push batch size when the push is rejected by Wavefront due to rate limit. Default: false. | true or false | false | |
| whitelistRegex | Regex pattern (java.util.regex) that input lines must match to be accepted. Input lines are checked against the pattern as they come in and before the prefix is prepended. | A valid regex pattern. | Allow points that begin with production. and stage.: ^(production|stage). |
3.1 |
| writeHttpJsonListenerPorts | Ports to listen on for incoming data from collectd write_http plugin. Default: None. | A comma-separated list of available port numbers. Can be a single port. | 4878 | 3.14 |
Data Buffering
If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:
2013-11-18 18:02:35,061 WARN [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]
By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:
2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared
Logging
The Wavefront proxy supports two log files: proxy log and blocked point log. To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. You configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.
Proxy Log
By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 30 log files, older files are deleted.
Blocked Point Log
You can log raw blocked points in a separate log from the proxy log. Logging of blocked points is disabled by default. To enable logging block points, edit the log4j2 configuration file and uncomment the blocked points file appender:
<!--
<AppenderRef ref="BlockedPointsFile"/>
-->
By default, blocked point entries are logged to <wavefront_log_path>/wavefront-blocked-points.log and the block point log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.