You can configure proxies using a configuration file, and you can performing advanced installation management such as installing proxies in a container.
In addition to the proxy configuration properties discussed here you can also use proxy preprocessor rules. These rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.
Proxy Configuration Properties
The main Wavefront proxy configuration file is maintained in <wavefront_config_path>/wavefront.conf (<wf_config_path>/wavefront.conf). The configuration file offers many options for changing how the proxy processes your data. There are optional configuration files for rewriting metrics and parsing log data. The default values work well in many cases, but you can adjust them as needed. After changing a configuration option, restart the proxy service.
Paths
In this section, file paths use the following conventions and values:
<wavefront_config_path>- Linux -
/etc/wavefront/wavefront-proxy - Mac -
/usr/local/etc/wavefront/wavefront-proxy - Windows -
C:\Program Files (x86)\Wavefront\conf
- Linux -
<wavefront_log_path>- Linux -
/var/log/wavefront - Mac -
/usr/local/var/log/wavefront - Windows -
C:\Program Files (x86)\Wavefront
- Linux -
<wavefront_spool_path>- Linux -
/var/spool/wavefront-proxy - Mac -
/usr/local/var/spool/wavefront-proxy - Windows -
C:\Program Files (x86)\Wavefront\bin
- Linux -
General Proxy Properties and Examples
This section lists:
- General proxy configuration properties
- Metrics proxy configuration properties
- Tracing proxy configuration properties
See Histogram Configuration Properties for properties specific to histogram distributions.
| Property | Purpose | Format and Example | Since |
|---|---|---|---|
| agentMetricsPointTags | Point tags and their values to be passed along with ~agent./ metrics. Default: None. |
Comma-separated list of key-value pairs. Ex: dc=west,env=prod |
3.24 |
| blacklistRegex | Regex pattern (java.util.regex) that input lines must match to be filtered out. Input lines are checked against the pattern as they come in and before the prefix is prepended. | Valid regex pattern. Ex: Filter out points that begin with qa., development., or test.: ^(qa|development|test). |
3.1 |
| buffer | Location of buffer files for saving failed transmissions for retry. | Valid path on the local file system. Ex: <wf_spool_path>/buffer |
3.20 |
| customSourceTags | Point tag keys to use as 'source' if no 'source' or 'host' field is present. Default: fqdn, hostname. | Comma-separated list of point tag keys. Ex: fqdn, hostname |
3.14 |
| dataBackfillCutoffHours | The cut-off point for what is considered a valid timestamp for back-dated points. We do not recommend setting this value larger than 1 year unless backfilling or migrating historic data. Default: 8760 (1 year), so all points older than 1 year are rejected. | Positive integer. Ex: 8760 |
4.1 |
| deltaCounterPorts | Port on which to listen only for delta counter data. Other data format are rejected at this port. If you specify a delta counter port, it can accept both HTTP and TCP data. For HTTP data, make a POST to this proxy port with an empty header, and the line-terminated data. Use this property in conjunction with deltaCounterAggregationInterval to limit the number of points per second for delta counters. Default: 50000. | Comma-separated list of available port numbers. Can be a single port. Ex: 50000 Ex: 50000,50001,50002 |
5.1 |
| deltaCounterAggregationInterval | Delay time for delta counter reporter. Use this property in conjunction with deltaCounterPorts to send points to the port(s) in batches, thereby limiting the number of points per second. Default: 30 seconds. | Number of seconds. Ex: 45 |
3.14 |
| ephemeral | Whether to automatically clean up old and orphaned proxy instances from the Wavefront Proxies page. We recommend enabling ephemeral mode if you're running the proxy in a container that may be frequently spun down and recreated. Default: false. | Boolean Ex: true |
3.14 |
| fileBeatPort | TCP port to listen on for Filebeat data. Default: 5044. | A port number. Ex: 5044 |
4.1 |
| flushThreads | Number of threads that flush data to the server. Setting this value too high results in sending batches that are too small to the Wavefront server and wasting connections. Values between 6 and 16 are a good starting point. This setting is per listening port. Default: The number of available processors (min 4). | Positive integer. Ex: 16 |
3.14 |
| graphiteDelimiters | Characters that should be replaced by dots, in case they were escaped within Graphite and collectd before sending. A common delimiter is the underscore character; so if you extract a hostname field with the value web04_www, it is changed to web04.www. |
A concatenation of delimiter characters, without any separators. | |
| graphiteFormat | Indexes of fields within Graphite and collectd metric names that correspond to a hostname. For example, if your metrics have the format: collectd.prod.www04.cpu.loadavg.1m, specify the 3rd and 2nd indexes (www04.prod) to be extracted and treated as the hostname. The remainder collectd.cpu.loadavg.1m is treated as the metric name. | Comma-separated list of indexes. Ex: 4, 2, 5 Ex: 3 |
|
| graphitePorts | TCP ports to listen on for Graphite data. Define which of the segments in your Graphite metrics map to a hostname in the graphiteFormat property. Default: 2003. | Comma-separated list of available port numbers. Can be a single port. Ex: 2003 Ex: 2003, 2004 |
|
| Histogram Configuration Properties | Properties specific to histogram distributions. | 4.31 | |
| hostname | A name unique across your account representing the machine that the proxy is running on. The hostname is not used to tag your metrics; rather, it's used to tag proxy metrics, such as JVM statistics, per-proxy point rates, and so on. | A string containing alphanumeric characters and periods. | |
| httpConnectTimeout | HTTP connect timeout (in milliseconds). Default: 5000 (5s). | Positive integer.
Ex: 5000 |
4.1 |
| httpRequestTimeout | HTTP request timeout (in milliseconds). We do not recommend setting this value to be higher than 20000. Recommended value for most configurations is 10000 (10 seconds). Default: 10000 (10s). | Positive integer.
Ex: 10000 |
4.1 |
| httpUserAgent | Override User-Agent in request headers. Can help bypass excessively restrictive filters on the HTTP proxy. Default user agent: Wavefront-Proxy/<version>. | A string.
Ex: 'Mozilla/5.0' |
4.1 |
| idFile | Location of the PID file for the wavefront-proxy process. Default: <cfg_path>/.wf_id. | Valid path on the local file system. | |
| jsonListenerPorts | TCP ports to listen on for incoming JSON-formatted metrics. Default: None. | Comma-separated list of available port numbers. Can be a single port. | |
| listenerIdleConnectionTimeout | Close idle inbound connections after specified time in seconds. Default: 300 | Number of seconds. | 4.31 |
| logsIngestionConfigFile | The file containing instructions for parsing log data into metrics. See Log Data Metrics Integration. Default: <cfg_path>/logsIngestion.yaml. | Valid path on the local file system. | 4.1 |
| opentsdbPorts | TCP ports to listen on for incoming OpenTSDB-formatted data. Default: None. | Comma-separated list of available port numbers. Can be a single port.
Ex: 4242 |
3.1 |
| picklePorts | TCP ports to listen on for incoming data in Graphite pickle format (from carbon-relay). Default: None. | Comma-separated list of available port numbers. Can be a single port.
Ex: 5878 |
3.20 |
| prefix | String to prepend before every metric name. For example, if you set prefix to 'production', a metric that is sent to the proxy as cpu.loadavg.1m is sent from the proxy to Wavefront as production.cpu.loadavg.1m. You can include longer prefixes such as production.nyc.dc1. Default: None. |
A lowercase alphanumeric string, with periods separating segments. You do not need to include a trailing period.
Ex: production
Ex: production.nyc.dc1
|
|
| preprocessorConfigFile | Path to the optional preprocessor config file containing preprocessor rules for filtering and rewriting metrics. Default: None. | Valid path on the local file system.
Ex: <cfg_path>/rules.yaml |
4.1 |
| proxyHost | HTTP proxy host to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyPort. | A string.
Ex: proxy.local |
3.23 |
| proxyPassword | When used with proxyUser, sets credentials to use with the HTTP proxy if the proxy requires authentication. | A string.
Ex: validPassword123 |
3.23 |
| proxyPort | HTTP proxy port to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyHost. | A port number.
Ex: 8080 |
3.23 |
| proxyUser | When used with proxyPassword, sets credentials to use with the HTTP proxy if the proxy requires authentication. | A string.
Ex: validUser |
3.23 |
| pushBlockedSamples | Number of blocked points to print to the log immediately following each summary line (every 10 flushes). If 0, print none. If you see a non-zero number of blocked points in the summary lines and want to debug what that data is, set this property to 5. Default: 0. | 0 or a positive integer.
Ex: 5 |
|
| pushFlushInterval | Milliseconds to wait between each flush to Wavefront. Default: 1000. | An integer equal to or greater than 1000.
Ex: 1000 |
|
| pushFlushMaxPoints | Maximum number of points to send to Wavefront during each flush. Default: 40,000. | Positive integer.
Ex: 40000 |
|
| pushListenerHttpBufferSize | Maximum allowed request size (in bytes) for incoming HTTP requests on Wavefront, OpenTSDB, or Graphite ports. Default: 16777216 (16MB). | Ex: 8388608 | 4.31 |
| pushListenerMaxReceivedLength | Maximum line length for received points in plaintext format on Wavefront, OpenTSDB, or Graphite ports. Default: 4096 | Positive integer.
Ex: 4096 |
4.31 |
| pushListenerPorts | Port to listen on for incoming data. A single port definition can accept both HTTP and TCP data. For HTTP data, make a POST to this proxy port with an empty header, and the line terminated data format. Default: 2878. | Comma-separated list of available port numbers. Can be a single port.
Ex: 2878
Ex: 2878,2879,2880 |
|
| pushLogLevel | Frequency to print status information on the data flow to the log. SUMMARY prints a line every 60 flushes, while DETAILED prints a line on each flush. | None, SUMMARY, or DETAILED
Ex: SUMMARY |
|
| pushMemoryBufferLimit | Maximum number of points that can stay in memory buffers before spooling to disk. Setting this value lower than default reduces memory usage but forces the proxy to queue points by spooling to disk more frequently, if you have points arriving at the proxy in short bursts. Default: 16 * pushFlushMaxPoints. Minimum: pushFlushMaxPoints. | Positive integer.
Ex: 640000 |
4.1 |
| pushRateLimit | Maximum number of points per second to send to Wavefront. Default: unlimited. | Positive integer.
Ex: 20000 |
4.1 |
| pushValidationLevel | Level of validation to perform on incoming data before sending the data to Wavefront. If NO_VALIDATION, all data is sent forward. If NUMERIC_ONLY, data is checked to make sure that it is numerical and dropped locally if it is not. | NUMERIC_ONLY or NO_VALIDATION
Ex: NUMERIC_ONLY |
|
| rawLogsMaxReceivedLength | Maximum line length for received raw logs. Default: 4096. | Positive integer.
Ex: 4096 |
4.31 |
| rawLogsPort | TCP port to listen on for log data. Default: 5045. | A port number.
Ex: 5045 |
4.4 |
| retryBackoffBaseSeconds | For exponential back-off when retry threads are throttled, the base (a in a^b) in seconds. Default: 2.0. | Positive number, integer or decimal.
Ex: 2.0 |
|
| retryThreads | Number of threads retrying failed transmissions. If no value is specified, defaults to the number of processor cores available to the host or 4, whichever is greater. Every retry thread uses a separate buffer file (capped at 2GB) to persist queued data points, so the number of threads controls the maximum amount of space that the proxy can use to buffer points locally. | Positive integer.
Ex: 4 |
|
| server | The API URL of the Wavefront server in the format https://<wf_instance>.wavefront.com/api/. | ||
| soLingerTime | Enable SO_LINGER with the specified linger time in seconds. Set this value to 0 when running in a high-availability configuration under a load balancer. Default: 0 (disabled). | 0 or a positive integer.
Ex: 0 |
4.1 |
| splitPushWhenRateLimited | Whether to split the push batch size when the push is rejected by Wavefront due to rate limit. Default: false. | true or false
Ex: false |
|
| whitelistRegex | Regex pattern (java.util.regex). Input lines are checked against the pattern as they come in and before the prefix is prepended. Only input lines that match are accepted. | Valid regex pattern.
Ex: ^(production|stage).
Allows points that begin with production. and stage. |
3.1 |
| writeHttpJsonListenerPorts | Ports to listen on for incoming data from the collectd write_http plugin. Default: None. | Comma-separated list of available port numbers. Can be a single port.
Ex: 4878 |
3.14 |
Tracing Proxy Properties and Examples
| Property | Purpose | Format /Example | Since |
|---|---|---|---|
| traceJaegerListenerPorts | TCP ports to listen on for Jaeger Thrift formatted data. Default: None. | Comma-separated list of available port numbers. Can be a single port. | 4.31 |
| traceListenerPorts | TCP ports to listen on for incoming trace data. Default: None. | Comma-separated list of available port numbers. Can be a single port.
Ex: 30000
Ex: 30000, 30001 |
4.31 |
| traceSamplingDuration | Minimum duration of the tracing spans that can be sent to Wavefront for trace data sampling. Default: 0 (send all generated spans). | Number of milliseconds.
Ex: 45 |
4.34 |
| traceSamplingRate | Percentage of all generated spans to send to Wavefront for trace data sampling. Default: 1.0 (send all generated spans). | Number from 0.0 to 1.0.
Ex: .1 |
4.34 |
| traceZipkinListenerPorts | TCP ports to listen on for Zipkin formatted data. Recommended: The default Zipkin Collector port (9411). Default: None. | Comma-separated list of available port numbers. Can be a single port. | 4.35 |
| traceJaegerApplicationName | Custom application name for traces received on Jaeger's traceJaegerListenerPorts. | traceJaegerApplicationName=MyJaeDemo | 4.38 |
| traceZipkinApplicationName | Custom application name for traces received on Zipkin's traceZipkinListenerPorts. | traceZipkinApplicationName=MyZipDemo | 4.38 |
| traceDerivedCustomTagKeys | Comma separated list of custom tag keys to include as metric tags for the derived RED(Request, Error, Duration) metrics. Applicable to Jaeger and Zipkin integration only. | traceDerivedCustomTagKeys=tenant, env, location | 4.38 |
Data Buffering
If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:
2013-11-18 18:02:35,061 WARN [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]
By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:
2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared
Logging
The Wavefront proxy supports two log files: proxy log and blocked point log.
To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. You configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.
If you’re using proxies in containers, you can mount the file, as discussed below.
Proxy Log
By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.
Blocked Point Log
You can log raw blocked points in a separate log from the proxy log. Logging of blocked points is disabled by default. To enable logging block points, edit the log4j2 configuration file and uncomment the blocked points file appender:
<!--
<AppenderRef ref="BlockedPointsFile"/>
-->
By default, blocked point entries are logged to <wavefront_log_path>/wavefront-blocked-points.log and the block point log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.
Configuring a Proxy in a Container
You can use the in-product Docker with cAdvisor or Kubernetes integration if you want to set up a proxy in a container. You can then customize that proxy.
Proxy Versions for Containers
For containers, the proxy image version is determined by the image property in the configuration file. You can set this to image: wavefronthq/proxy:latest, or specify a proxy version explicitly.
The proxies are not stateful. Your configuration is managed in your yaml file. It’s safe to use proxy:latest – we ensure that proxies are backward compatible.
Restricting Memory Usage for the Container
To restrict memory usage of the container using Docker, you need to add a JAVA_HEAP_USAGE environment variable and restrict memory using the -m or --memory options for the docker run command. The container memory contraint should be at least 350mb larger than the JAVA_HEAP_USAGE environment variable.
To restrict a container’s memory usage to 2g with Docker run:
docker run -d --name wavefront-proxy ... -e JAVA_HEAP_USAGE="1650m" -m 2g ...
To limit memory usage of the container in Kubernetes use the resources.limits.memory property of a container definition. See the Kubernetes doc
Customizing Proxy Settings for Docker
When you run a Wavefront proxy inside a Docker container, you can tweak proxy configuration settings that are properties in the wavefront.conf file directly from the Docker run command. You use the WAVEFRONT_PROXY_ARGS environment variable and pass in the property name as a long form argument, preceded by --.
For example, add -e WAVEFRONT_PROXY_ARGS="--pushRateLimit 1000" to your docker run command to specify a rate limit of 1000 pps for the proxy.
See the Wavefront Proxy configuration file for a full list.
Logging Customization for Containers
You can customize logging by mounting a customized log4j2.xml file. Here’s an example for Docker:
--mount type=bind, src=<absolute_path>/log4j2.xml, dst=/etc/wavefront/wavefront-proxy/log4j2.xml
See Logging above for additional background.
Installing Proxies on Multiple Linux Hosts
Ansible is an open-source automation engine that automates software provisioning, configuration and management, and application deployment. The Wavefront Ansible role installs and configures the Wavefront proxy, which allows you to automate Wavefront proxy installation on multiple Linux hosts.
Note: In most cases, you install only one or two proxies in your environment. You don’t need a proxy for each host you collect data from. See Proxy Deployment Options.
For details, see the Setup tab in the Ansible built-in integration.