Even without additional customization the Wavefront proxy ingests metrics and forwards them to the Wavefront service in a secure, fast, and reliable manner. If needed, you can customize your proxy.
- Proxy configuration properties allow you to changing how the proxy processes your data. For example, you can change ports or perform other advanced installation management.
- Proxy preprocessor rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.
- Log files can help in case of problems.
Proxy File Paths
By default, proxy files are installed in the following locations.
- Configuration properties:
<wavefront_config_path>
- Linux -
/etc/wavefront/wavefront-proxy
- Mac -
/usr/local/etc/wavefront/wavefront-proxy
- Windows -
C:\Program Files (x86)\Wavefront\conf
- Linux -
- Parse log data:
<wavefront_log_path>
- Linux -
/var/log/wavefront
- Mac -
/usr/local/var/log/wavefront
- Windows -
C:\Program Files (x86)\Wavefront
- Linux -
<wavefront_spool_path>
- Linux -
/var/spool/wavefront-proxy
- Mac -
/usr/local/var/spool/wavefront-proxy
- Windows -
C:\Program Files (x86)\Wavefront\bin
- Linux -
Data Buffering
If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log
:
2013-11-18 18:02:35,061 WARN [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]
By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log
:
2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared
If you don’t want to buffer the data on a file-based storage and if you have an AWS Simple Queue Service (SQS), you can add an SQS for the proxy so that the data is sent to the SQS instead of buffering the data to the local on-disk when there is a data outage or when proxies are backing up. To send data to an AWS SQS, configure the
sqsBuffer
, sqsQueueNameTemplate
, sqsQueueIdentifier
, and sqsQueueRegion
properties in the wavefront.conf
file.Logging
The Wavefront proxy supports two log files: proxy log and blocked point log.
To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. Configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml
. For details on log4j2 configuration, see Log4j Configuration.
If you’re using proxies in containers, you can mount the log files, as discussed below.
Proxy Log
By default, proxy log entries are logged to <wavefront_log_path>
/wavefront.log
. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.
If you want to set logs for Jaeger and Zipkin integrations, see Logging for Jaeger and Zipkin.
Blocked Data Log
You can log all the raw blocked data separately or log different entities into their separate log files.
- Log the block data separately
Follow these steps:- Open the
<wavefront_config_path>
/log4j2.xml
configuration file. - To log all the block data, uncomment the corresponding section.
<AsyncLogger name="RawBlockPoints" level="WARN" additivity="false"> <AppenderRef ref="BlockedPointsFile" /> </AsyncLogger>
By default, blocked point entries are logged to the
<wavefront_log_path>/wavefront-blocked-points.log
file and the log file is rolled over every day when its size reaches 100MB. When there are 31 log files, older files are deleted. You can customize the configurations to suit your environment.
- Open the
- Set up separate log files for blocked entities
Follow these steps:- Uncomment or add the configurations under Appenders and Loggers in the
<wavefront_config_path>
/log4j2.xml
configuration file.<!-- Log the blocked histograms. If you don't need a separate log file for it, don't add this configuration to the file.--> <AsyncLogger name="RawBlockedHistograms" level="WARN" additivity="false"> <AppenderRef ref="[Enter_Your_File_Name]"/> </AsyncLogger> <!-- Logs the blocked points for spans. If you don't need a separate log file for it, don't add this configuration to the file.--> <AsyncLogger name="RawBlockedSpans" level="WARN" additivity="false"> <AppenderRef ref="[Enter_Your_File_Name]"/> </AsyncLogger> <AsyncLogger name="RawBlockPoints" level="WARN" ADDITIVITY="FALSE"/> <AppenderRef ref=”BlockedPointsFile”/> </AsyncLogger>
- Add the names of the block points, which you uncommented in the
log4j2.xml
file, to the<wavefront_config_path>
/wavefront.conf
file.
Example:blockedPointsLoggerName = RawBlockedPoints # Add this if you added the appender for histograms in the log4j2.xml file. blockedHistogramsLoggerName = RawBlockedHistograms (RawBlockedPoints by default) # Add this if you added the appender for spans in the log4j2.xml file. blockedSpansLoggerName = RawBlockedSpans (RawBlockedPoints by default)
Tip: You must update both the<wavefront_log_path>/log4j2.xml
file and the<wavefront_config_path>/wavefront.conf
file to get separate log files for blocked entities. - Uncomment or add the configurations under Appenders and Loggers in the
Configuration Properties
This section gives details on proxy configuration properties. All properties are also listed, in the wavefront.conf.default file on Github.
General Configuration Properties
Property | Purpose | Format and Example | Since |
---|---|---|---|
agentMetricsPointTags | Point tags and their values to be passed along with ~agent./ metrics. Default: None. |
Comma-separated list of key-value pairs. Ex: dc=west,env=prod |
3.24 |
block | Regex pattern (java.util.regex) that input lines must match to be filtered out. Input lines are checked against the pattern as they come in and before the prefix is prepended. Renamed from blackListRegex to block in proxy 9.x. Use preprocessor rules for finer-grained control. |
Valid regex pattern. Ex: Filter out points that begin with qa., development., or test.: ^(qa|development|test). |
3.1/9.x |
blockedPointsLoggerName | Controls how the blocked points are logged. For details, see Blocked Data Log. Default: RawBlockedPoints. |
Logger name for blocked points. Ex: RawBlockedPoints |
6.0 |
blockedHistogramsLoggerName | Controls how the blocked histograms are logged. For details, see Blocked Data Log. Default: RawBlockedPoints. |
Logger name for blocked points. Ex: RawBlockedPoints |
6.0 |
blockedSpansLoggerName | Controls how the blocked spans are logged. For details, see Blocked Data Log. Default: RawBlockedPoints. |
Logger name for blocked points. Ex: RawBlockedPoints |
6.0 |
buffer | Location of buffer files for saving failed transmissions for retry. | Valid path on the local file system. Ex: <wf_spool_path>/buffer |
The start |
customSourceTags | Point tag keys to use as 'source' if no 'source' or 'host' field is present. Default: fqdn, hostname. |
Comma-separated list of point tag keys. Ex: fqdn, hostname |
3.14 |
dataPrefillCutoffHours | Defines the cut-off point for what is considered a valid timestamp for pre-dated points. Default (and recommended) value is 24 (1 day), so all the data points from more than 1 day in future are rejected. | Positive integer. Ex: 12 |
4.1 |
dataBackfillCutoffHours | The cut-off point for what is considered a valid timestamp for back-dated points. We do not recommend setting this value larger than 1 year unless backfilling or migrating historic data. Default: 8760 (1 year), so all points older than 1 year are rejected. |
Positive integer. Ex: 8760 |
4.1 |
deltaCountersAggregationListenerPorts | Port to listen to the Wavefront-formatted delta counter data. Other data formats are rejected at this port. Pre-aggregating delta counters at the proxy, helps reduce the outbound point rate. Use this property in conjunction with deltaCountersAggregationIntervalSeconds to limit the number of points per second for delta counters. Default: none. |
Comma-separated list of available port numbers. Can be a single port. Ex: 12878 Ex: 12878,12879 |
6.0 |
deltaCountersAggregationIntervalSeconds | Interval between flushing aggregating delta counters to Wavefront. Use this property in conjunction with deltaCountersAggregationListenerPorts to send points to the port(s) in batches, thereby limiting the number of points per second. Default: 30 seconds. |
Number of seconds. Ex: 45 |
6.0 |
fileBeatPort | TCP port to listen on for Filebeat data. Default: none. |
A port number. Ex: 5044 |
4.1 |
flushThreads | Number of threads that flush data to the server. Setting this value too high results in sending batches that are too small to the Wavefront server and wasting connections. Values between 6 and 16 are a good starting point. This setting is per listening port. Default: The number of available processors (min 4). |
Positive integer. Ex: 16 |
3.14 |
graphiteDelimiters | Characters that should be replaced by dots, in case they were escaped within Graphite and collectd before sending. A common delimiter is the underscore character; so if you extract a hostname field with the value web04_www , it is changed to web04.www . |
A concatenation of delimiter characters, without any separators. | |
graphiteFormat | Indexes of fields within Graphite and collectd metric names that correspond to a hostname. For example, if your metrics have the format: collectd.prod.www04.cpu.loadavg.1m, specify the 3rd and 2nd indexes (www04.prod) to be extracted and treated as the hostname. The remainder collectd.cpu.loadavg.1m is treated as the metric name. | Comma-separated list of indexes. Ex: 4, 2, 5 Ex: 3 |
|
graphitePorts | TCP ports to listen on for Graphite data. Define which of the segments in your Graphite metrics map to a hostname in the graphiteFormat property. Default: None. | Comma-separated list of available port numbers. Can be a single port. Ex: 2003 Ex: 2003, 2004 |
|
gzipCompression | If set to true, metric traffic from the proxy to the Wavefront endpoint is gzip-compressed. Default: true. |
true or false Ex: true |
|
gzipCompressionLevel | Sets the gzip compression level if gzipCompression is enabled. The level vary from, 1 to 9. Higher compression levels slightly reduces the volume of traffic between the proxy and Wavefront, but uses more CPU. Default: 4. |
Positive integer ranging from 1 to 9. Ex: 4 |
6.0 |
Histogram configuration properties | Properties specific to histogram distributions, listed in a separate table below. | 4.31 | |
hostname | A name unique across your account representing the machine that the proxy is running on. The hostname is not used to tag your metrics; rather, it's used to tag proxy metrics, such as JVM statistics, per-proxy point rates, and so on. | A string containing alphanumeric characters and periods. | |
httpConnectTimeout | HTTP connect timeout (in milliseconds). Default: 5000 (5s). |
Positive integer.
Ex: 5000 |
4.1 |
httpRequestTimeout | HTTP request timeout (in milliseconds). We do not recommend setting this value to be higher than 20000. Recommended value for most configurations is 10000 (10 seconds). Default: 10000 (10s). |
Positive integer.
Ex: 10000 |
4.1 |
httpMaxConnTotal | Max number of total connections to keep open. Default: 200 |
Positive integer.
Ex: 100 |
|
httpMaxConnPerRoute | Max number of connections per route to keep open. Default: 100. |
Positive integer.
Ex: 50 |
|
httpUserAgent | Override User-Agent in request headers. Can help bypass excessively restrictive filters on the HTTP proxy. Default user agent: Wavefront-Proxy/<version>. | A string.
Ex: 'Mozilla/5.0' |
4.1 |
idFile | Location of the PID file for the wavefront-proxy process. Default: <dshell>/.id |
Valid path on the local file system. This option is ignored when ephemeral=true. | |
jsonListenerPorts | TCP ports to listen on for incoming JSON-formatted metrics. Default: None. |
Comma-separated list of available port numbers. Can be a single port. | |
listenerIdleConnectionTimeout | Close idle inbound connections after specified time in seconds. Default: 300 |
Number of seconds. | 4.31 |
logsIngestionConfigFile | The file containing instructions for parsing log data into metrics. See Log Data Metrics Integration. Default: <cfg_path>/logsIngestion.yaml. | Valid path on the local file system. | 4.1 |
opentsdbPorts | TCP ports to listen on for incoming OpenTSDB-formatted data. Default: None. |
Comma-separated list of available port numbers. Can be a single port.
Ex: 4242 |
3.1 |
picklePorts | TCP ports to listen on for incoming data in Graphite pickle format (from carbon-relay). Default: None. |
Comma-separated list of available port numbers. Can be a single port.
Ex: 5878 |
3.20 |
prefix | String to prepend before every metric name. For example, if you set prefix to 'production', a metric that is sent to the proxy as cpu.loadavg.1m is sent from the proxy to Wavefront as production.cpu.loadavg.1m . You can include longer prefixes such as production.nyc.dc1 . Default: None. |
A lowercase alphanumeric string, with periods separating segments. You do not need to include a trailing period.
Ex: production
Ex: production.nyc.dc1
|
|
preprocessorConfigFile | Path to the optional preprocessor config file containing preprocessor rules for filtering and rewriting metrics. Default: None. |
Valid path on the local file system.
Ex: <cfg_path>/rules.yaml |
4.1 |
privateKeyPath | Path to PKCS#8 private key file in PEM format. Incoming TLS/SSL connections access this private key. | 8.0 | |
privateCertPath | Path to X.509 certifivate chain file in PEM format. Incoming TLS/SSL connections access this certificate. | 8.0 | |
proxyHost | HTTP proxy host to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyPort. | A string.
Ex: proxy.local |
3.23 |
proxyPassword | When used with proxyUser, sets credentials to use with the HTTP proxy if the proxy requires authentication. | A string.
Ex: validPassword123 |
3.23 |
proxyPort | HTTP proxy port to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyHost. | A port number.
Ex: 8080 |
3.23 |
proxyUser | When used with proxyPassword, sets credentials to use with the HTTP proxy if the proxy requires authentication. | A string.
Ex: validUser |
3.23 |
pushBlockedSamples | Number of blocked points to print to the log immediately following each summary line (every 10 flushes). If 0, print none. If you see a non-zero number of blocked points in the summary lines and want to debug what that data is, set this property to 5. Default: 5 Note: Defaults to 5 since version 5.0. Before version 5.0 it defaulted to 0.
|
0 or a positive integer.
Ex: 5 |
|
pushFlushInterval | Milliseconds to wait between each flush to Wavefront. Default: 1000. |
An integer equal to or greater than 1000.
Ex: 1000 |
|
pushFlushMaxPoints | Maximum number of points to send to Wavefront during each flush. Default: 40,000. |
Positive integer.
Ex: 40000 |
|
pushFlushMaxHistograms | Maximum number of histograms to send to Wavefront during each flush. Default: 10,000. |
Positive integer.
Ex: 10000 |
6.0 |
pushFlushMaxSpans | Maximum number of spans to send to Wavefront during each flush. Default: 5,000. |
Positive integer.
Ex: 5000 |
6.0 |
pushFlushMaxSpanLogs | Maximum number of span logs to send to Wavefront during each flush. Default: 1,000. |
Positive integer.
Ex: 1000 |
6.0 |
pushListenerHttpBufferSize | Maximum allowed request size (in bytes) for incoming HTTP requests on Wavefront, OpenTSDB, or Graphite ports. Default: 16777216 (16MB). |
Ex: 8388608 | 4.31 |
pushListenerMaxReceivedLength | Maximum line length for received points in plaintext format on Wavefront, OpenTSDB, or Graphite ports. Default: 32KB |
Positive integer.
Ex: 4096 |
4.31 |
pushListenerPorts | Port to listen on for incoming data. A single port definition can accept both HTTP and TCP data. For HTTP data, make a POST to this proxy port with an empty header, and the line terminated data format. If you want to use HTTPS/TLS, set the tlsPort, privateKeyPath, and privateCertPath as well. Default: 2878. |
Comma-separated list of available port numbers. Can be a single port.
Ex: 2878
Ex: 2878,2879,2880 |
|
pushLogLevel | Frequency to print status information on the data flow to the log. SUMMARY prints a line every 60 flushes, while DETAILED prints a line on each flush.
Warning: This configuration was deprecated in version 6.0. The level of logging is controlled through log4j2 configurations.
|
None, SUMMARY, or DETAILED
Ex: SUMMARY |
Before 6.0. |
pushMemoryBufferLimit | Maximum number of points that can stay in memory buffers before spooling to disk. Setting this value lower than default reduces memory usage but forces the proxy to queue points by spooling to disk more frequently, if you have points arriving at the proxy in short bursts. Default: 16 * pushFlushMaxPoints. Minimum: pushFlushMaxPoints. | Positive integer.
Ex: 640000 |
4.1 |
pushRateLimit | Maximum number of points per second to send to Wavefront. Default: unlimited. |
Positive integer.
Ex: 20000 |
4.1 |
pushRateLimitHistograms | Maximum number of histograms per second to send to Wavefront. Default: unlimited. |
Positive integer.
Ex: 20000 |
6.0 |
pushRateLimitSpans | Maximum number of spans per second to send to Wavefront. Default: unlimited. |
Positive integer.
Ex: 10000 |
6.0 |
pushRateLimitSpanLogs | Maximum number of span logs per second to send to Wavefront. Default: unlimited. |
Positive integer.
Ex: 10000 |
6.0 |
pushRateLimitMaxBurstSeconds | Max number of burst seconds to allow when rate limiting to smooth out uneven traffic. Set to 1 when doing data backfills. Default: 10 |
Positive integer.
Ex: 20 |
|
pushRelayListenerPorts | Ports to receive the data sent to the relay. In environments where direct outbound connections to Wavefront servers are not possible, you can use another Wavefront proxy that has outbound access to act as a relay and forward all the data received on that endpoint (from direct data ingestion clients and/or other proxies) to Wavefront servers. Default: none. |
Comma-separated list of available port numbers. Can be a single port.
Ex: 2978 Ex: 2978,2979 |
6.0 |
pushRelayHistogramAggregator | If set to true, aggregates the histogram distributions received on the relay port. Default: false. |
true or false
Ex: true |
6.0 |
pushValidationLevel | Level of validation to perform on incoming data before sending the data to Wavefront. If NO_VALIDATION, all data is sent forward. If NUMERIC_ONLY, data is checked to make sure that it is numerical and dropped locally if it is not. | NUMERIC_ONLY or NO_VALIDATION
Ex: NUMERIC_ONLY |
|
rawLogsMaxReceivedLength | Maximum line length for received raw logs. Default: 4096. |
Positive integer.
Ex: 4096 |
4.31 |
rawLogsPort | TCP port to listen on for log data. Default: none. |
A port number.
Ex: 5045 |
4.4 |
retryBackoffBaseSeconds | For exponential back-off when retry threads are throttled, the base (a in a^b) in seconds. Default: 2.0. |
Positive number, integer or decimal.
Ex: 2.0 |
|
retryThreads | Number of threads retrying failed transmissions. If no value is specified, defaults to the number of processor cores available to the host or 4, whichever is greater. Every retry thread uses a separate buffer file (capped at 2GB) to persist queued data points, so the number of threads controls the maximum amount of space that the proxy can use to buffer points locally.
Warning: This configuration was deprecated in version 6.0 because we redesigned the storage engine to improve spooling data to disk.
|
Positive integer.
Ex: 4 |
Before 6.0. |
server | The API URL of the Wavefront server in the format https://<wf_instance>.wavefront.com/api/. | ||
soLingerTime | Enable SO_LINGER with the specified linger time in seconds. Set this value to 0 when running in a high-availability configuration under a load balancer. Default: -1 (disabled). |
0 or a positive integer.
Ex: 0 |
4.1 |
splitPushWhenRateLimited | Whether to split the push batch size when the push is rejected by Wavefront due to rate limit. Default: false. |
true or false
Ex: false |
|
tlsPorts | Comma-separated list of ports to be used for incoming TLS/SSL connections. To set up a port to use TLS/SSL, you specify pushListenerPorts, tlsPorts, privateKeyPath, and privateCertPath. | 8.0 | |
allow | Regex pattern (java.util.regex). Input lines are checked against the pattern as they come in and before the prefix is prepended. Only input lines that match are accepted. Renamed from whiteListRegex to allow in proxy 9.x. Use preprocessor rules for finer-grained control. |
Valid regex pattern.
Ex: ^(production|stage).
Allows points that begin with production. and stage. |
3.1 |
writeHttpJsonListenerPorts | Ports to listen on for incoming data from the collected write_http plugin. Default: None. |
Comma-separated list of available port numbers. Can be a single port.
Ex: 4878 |
3.14 |
rawLogsHttpBufferSize | The maximum request size (in bytes) for incoming HTTP requests with tracing data. Default: 16MB |
Buffer size in bytes. Ex: 16777216 |
4.38 |
trafficShaping | Enables intelligent traffic shaping based on the data receive rate over the last 5 minutes. Default: false |
true or false | 9.0 |
trafficShapingQuantile | Sets the quantile for traffic shaping.
Default: 75 |
An integer Ex: 99 The 99th percentile of the received rate in the last 5 minutes, will be used as a basis for the rate limiter. |
9.0 |
trafficShapingHeadroom |
Sets the headroom multiplier for traffic shaping when there's backlog.
Default: 1.15 (15% headroom) |
Number from 1.0 to 1.99 Ex: 1.05 (5% headroom) |
9.0 |
sqsBuffer | Use AWS SQS for buffering transmissions.
Default: false |
true or false | 9.0 |
sqsQueueNameTemplate | The replacement pattern for naming the SQS queues. | Ex: wf-proxy-{{id}}-{{entity}}-{{port}} results in a queue named wf-proxy-id-points-2878 |
9.0 |
sqsQueueIdentifier | An identifier for identifying the proxies in SQS. | A string Ex: wavefront |
9.0 |
sqsQueueRegion | The AWS Region name the queue lives in. | A string Ex: us-west-2 |
9.0 |
Authentication Proxy Properties
Because the proxy is running in your local network by default, communication to the proxy is un-authenticated. If you want to authenticate inbound traffic to the proxy, use the settings in this section.
The Wavefront proxy must authenticate outbound traffic to the Wavefront service. See Authenticate Incoming HTTP Requests at the Proxy for step-by-step instructions.
Property | Purpose | Format and Example |
---|---|---|
authMethod | Authentication is done by a token. Specify the tokens either in the query string ("token=" and "api_key=" parameters) or in the request headers ("X-AUTH-TOKEN: ", "Authorization: Bearer", "Authorization: " headers). TCP streams are disabled when authentication is turned on. Default: NONE. |
Allowed authMethod: NONE, STATIC_TOKEN, HTTP_GET, OAUTH2. Ex: OAUTH2 |
authTokenIntrospectionServiceUrl | URL for the token introspection endpoint used to validate tokens for incoming HTTP requests (for example, authTokenIntrospectionServiceUrl = https://auth.acme.de/api/token/<token>/validate ).
Required when authMethod is OAUTH2 or HTTP_GET Default: None. |
Valid URL |
authTokenIntrospectionAuthorizationHeader | Optional credentials for use with the token introspection endpoint if the endpoint requires authentication. For example:authTokenIntrospectionAuthorizationHeader=
Authorization: Bearer <token> |
|
authResponseRefreshInterval | Cache TTL (in seconds) for token validation results (re-authenticate when expired). Default: 600 seconds. |
Integer Ex: 1200 |
authResponseMaxTtl | Maximum allowed cache TTL (in seconds) for token validation results when the token introspection service is unavailable. Default: 86400 seconds (1 day). |
Integer Ex: 3600 |
authStaticToken | Static token that is considered valid for all incoming HTTP requests. Required when authMethod = STATIC_TOKEN. For example, authStaticToken=token1234abcd Default: None |
Valid token |
Tracing Proxy Properties
Property | Purpose | Format /Example |
---|---|---|
traceAlwaysSampleErrors | Always sample spans with an error tag ignoring other sampling configuration.
Default: true |
true or false
Ex: false |
traceJaegerHttpListenerPorts | TCP ports to receive Jaeger Thrift formatted data via HTTP. The data is then sent to Wavefront in Wavefront span format.
Default: None. Version: Since 6.0 |
Comma-separated list of available port numbers. Can be a single port. |
traceJaegerListenerPorts | TCP ports to receive Jaeger Thrift formatted data via TChannel. The data is then sent to Wavefront in Wavefront span format. Default: None. Warning:
Sending data via TChannel has been deprecated in Jaeger 1.16. Therefore, we recommend using traceJaegerHttpListenerPorts to receive Jaeger Thrift formatted data via HTTP. |
Comma-separated list of available port numbers. Can be a single port. |
traceJaegerGrpcListenerPorts | Ports to receive Jaeger Protobuf formatted data over gRPC.
Default: None. Version: Since 9.0 |
Comma-separated list of available port numbers. Can be a single port. |
customTracingListenerPorts | TCP ports to receive spans and derive RED metrics from the SDKs that send raw data to Wavefront.
Default: None. Version: Since 6.0 Note:
The application name and service name tags are required to generate RED metrics. If these tags are not sent with your span, the application name defaults to wfProxy , and the service name defaults to defaultService . |
Comma-separated list of available port numbers. Can be a single port. |
traceListenerMaxReceivedLength | Maximum line length for received spans and span logs.
Default: 1MB |
Integer
Ex: 1048576 |
traceListenerPorts | TCP ports that listen to incoming spans from the Wavefront SDKs that collect trace data, collect metrics and histograms, and Wavefront SDKs that instrument frameworks. Default: None. |
Comma-separated list of available port numbers. Can be a single port.
Ex: 30000
Ex: 30000, 30001 |
traceSamplingDuration | Minimum duration of the tracing spans that can be sent to Wavefront for trace data sampling. Default: 0 (send all generated spans). |
Number of milliseconds.
Ex: 45 |
traceSamplingRate | Percentage of all generated spans to send to Wavefront for trace data sampling. Default: 1.0 (send all generated spans). |
Number from 0.0 to 1.0.
Ex: .1 |
traceZipkinListenerPorts | TCP ports to listen on for Zipkin formatted data. Recommended: The default Zipkin Collector port (9411). Default: None. |
Comma-separated list of available port numbers. Can be a single port. |
customTracingApplicationName | Custom application name for spans received on the customTracingListenerPorts that don't have the application tag.
Default: defaultApp. Version: Since 9.0 |
customTracingApplicationName=MyApplication |
customTracingServiceName | Custom service name for spans received on the customTracingListenerPorts that don't have the service tag.
Default: defaultService. Version: Since 9.0 |
customTracingServiceName=MyService |
traceJaegerApplicationName | Custom application name for traces received on Jaeger's traceJaegerListenerPorts or traceJaegerHttpListenerPorts. | traceJaegerApplicationName=MyJaegerDemo |
traceZipkinApplicationName | Custom application name for traces received on Zipkin's traceZipkinListenerPorts. | traceZipkinApplicationName=MyZipkinDemo |
traceDerivedCustomTagKeys | Comma separated list of custom tag keys to include as metric tags for the derived RED (Request, Error, Duration) metrics. Applicable to Jaeger and Zipkin integration only. | traceDerivedCustomTagKeys=tenant, env, location |
traceListenerHttpBufferSize | The maximum request size (in bytes) for incoming HTTP requests with tracing data. Default: 16MB |
Buffer size in bytes. Ex: 16777216 |
Histogram Configuration Properties
Wavefront supports additional histogram configuration properties, shown in the following table. Note the requirements on the state directory and the effect of the two persist
properties listed at the bottom of the table.
Property | Description | Format |
---|---|---|
histogramAccumulatorResolveInterval | Interval in milliseconds to write back accumulation changes from memory cache to disk. Only applicable when memory cache is enabled. Increasing this setting reduces storage IO pressure but might increase heap memory use. Default: 100. | Positive integer. |
histogramAccumulatorFlushInterval | Interval in milliseconds to check for histograms that need to be sent to Wavefront according to their histogramMinuteFlushSecs settings. Default: 1000. | Positive integer. |
histogramAccumulatorFlushMaxBatchSize | Max number of histograms to move to the outbound queue in one flush. Default: no limit. | Positive integer. |
histogramDayAccumulators | Number of accumulators per day port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. | Positive integer. |
histogramDayAccumulatorSize | Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. | Positive integer. |
histogramDayAccumulatorPersisted | Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. | Boolean. Ex: false |
histogramDayAvgDigestBytes | Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramDayCompression * 7 | Positive integer. |
histogramDayAvgKeyBytes | Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. | Positive integer. |
histogramDayCompression | A bound on the number of centroids per histogram. Default: 100. | Positive integer in the interval [20;1000]. |
histogramDayFlushSecs | Time-to-live, in seconds, for a day granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 18000 (5 hours). | Positive integer. |
histogramDayListenerPorts | TCP ports to listen on for histograms to be aggregated by day. Default: 40003. | Comma-separated list of ports. |
histogramDayMemoryCache | Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. | Boolean. |
histogramDistAccumulators | Number of accumulators per distribution port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: number of available CPU cores. | Positive integer. |
histogramDistAccumulatorPersisted | Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. | Boolean. Ex: false |
histogramDistAccumulatorSize | Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. | Positive integer. |
histogramDistAvgDigestBytes | Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramDistCompression * 7 | Positive integer. |
histogramDistCompression | A bound on the number of centroids per histogram. Default: 100. | Positive integer in the interval [20;1000]. |
histogramDistFlushSecs | Number of seconds to keep a new distribution bin open for new samples, before the intermediary is sent to Wavefront. Default: 70. | Positive integer. |
histogramDistListenerPorts | TCP ports to listen on for ingesting histogram distributions. Default: 40000. | Comma-separated list of ports. Can be a single port. |
histogramDistMemoryCache | Enabling memory cache reduces I/O load with fewer time series and higher frequency data (Aggregating more than 1 distribution per second per time series). Default: false. | Boolean. |
histogramDistAvgKeyBytes | Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. | Positive integer. |
histogramHttpBufferSize | Since 4.40. The maximum request size (in bytes) for incoming HTTP requests on histogram ports. Default: 16MB. |
Buffer size in bytes. Ex: 16777216 |
histogramHourAccumulators | Number of accumulators per hour port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. | Positive integer. |
histogramHourAccumulatorSize | Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. | Positive integer. |
histogramHourAccumulatorPersisted | Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. | Boolean. Ex: false |
histogramHourAvgDigestBytes | Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramMinuteCompression * 7 | Positive integer. |
histogramHourAvgKeyBytes | Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. | Positive integer. |
histogramHourCompression | A bound on the number of centroids per histogram. Default: 100. | Positive integer in the interval [20;1000]. |
histogramHourFlushSecs | Time-to-live, in seconds, for an hour granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 4200. | Positive integer. |
histogramHourListenerPorts | TCP ports to listen on for histograms to be aggregated by hour. Default: 40002. | Comma-separated list of ports. Can be a single port. |
histogramHourMemoryCache | Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. | Boolean. |
histogramMaxReceivedLength | Maximum line length for received histogram points. Default: 65536. | Positive integer. |
histogramMinuteAccumulatorPersisted | Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. | Boolean. Ex: false |
histogramMinuteListenerPorts | TCP ports to listen on for histograms to be aggregated by minute. Default: 40001. | Comma-separated list of ports. Can be a single port. |
histogramMinuteAccumulators | Number of accumulators per minute port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. | Positive integer. |
histogramMinuteAccumulatorSize | Expected upper bound of concurrent accumulations. Should be approximately the number of timeseries * 2 (use a higher multiplier if out-of-order points more than 1 minute apart are expected). Setting this value too high results in excessive disk space usage, setting this value too low may cause severe performance issues. Default: 100000. | Positive integer. |
histogramMinuteAvgKeyBytes | Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. | Positive integer. |
histogramMinuteCompression | A bound on the number of centroids per histogram. Default: 32. | Positive integer in the interval. [20;1000]. |
histogramMinuteFlushSecs | Time-to-live, in seconds, for a minute granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 70. | Positive integer. |
histogramMinuteMemoryCache | Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. | Boolean. |
histogramMinuteAvgDigestBytes | Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramMinuteCompression * 7 | Positive integer. |
histogramProcessingQueueScanInterval | Interval in milliseconds between checks for new entries in the processing queue. Default: 20. | Positive integer. |
histogramReceiveBufferFlushInterval | Sets maximum time in milliseconds that incoming points can stay in the receive buffer when incoming traffic volume is very low. Default: 100. | Positive integer. |
histogramStateDirectory | Directory for persistent proxy state, must be writable. Before being flushed to Wavefront, histogram data is persisted on the filesystem where the Wavefront proxy resides. If the files are corrupted or the files in the directory can't be accessed, the proxy reports the problem in its log and fails back to using in-memory structures. In this mode, samples can be lost if the proxy terminates without draining its queues. Default: /var/spool/wavefront-proxy .
|
A valid path on the local file system. Note: A high PPS requires that the machine that the proxy is on has an appropriate amount of IOPS. We recommend about 1K IOPS with at least 8GB RAM on the machine that the proxy writes histogram data to. Recommended machine type: m4.xlarge.
|
persistAccumulator | Whether to persist accumulation state. We suggest keeping this setting enabled unless you are not using hour and day level aggregation and consider losing up to 1 minute worth of data during proxy restarts acceptable. Default: true. | . Warning: If set to false, unprocessed metrics are lost on proxy shutdown.
|
persistMessages | Whether to persist received metrics to disk. Default: true. | Boolean. Warning: If set to false, unprocessed metrics are lost on proxy shutdown.
|
pushRelayHistogramAggregatorAccumulatorSize | Since 6.0. Max number of concurrent histogram to accumulations at the relay ports. The value is approximately the number of time series * 2 (use a higher multiplier if out-of-order points more than 1 bin apart are expected). Setting this value too high will cause excessive disk space usage, setting this value too low may cause severe performance issues. Only applicable if the pushRelayHistogramAggregator is set to true. Default: 32. |
Positive integer. |
pushRelayHistogramAggregatorCompression | Since 6.0. Number of centroids per histogram. Only applicable if the pushRelayHistogramAggregator is set to true. Default: 32. |
Must be between 20 and 1000. Ex: 40 |
pushRelayHistogramAggregatorFlushSecs | Since 6.0. Interval in milliseconds to check for histograms that have accumulated at the relay ports before sending data to Wavefront. Only applicable if the pushRelayHistogramAggregator is set to true. Default: 70. |
Number of milliseconds. Ex: 80 |
Configure a Proxy in a Docker Container
You can use the in-product Docker with cAdvisor or Kubernetes integration if you want to set up a proxy in a container. You can then customize that proxy.
Proxy Versions for Containers
For containers, the proxy image version is determined by the image
property in the configuration file. You can set this to image: wavefronthq/proxy:latest
, or specify a proxy version explicitly.
The proxies are not stateful. Your configuration is managed in your yaml
file. It’s safe to use proxy:latest
– we ensure that proxies are backward compatible.
Restrict Memory Usage for a Docker Container
To restrict memory usage of the container using Docker, you need to add a JAVA_HEAP_USAGE
environment variable and restrict memory using the -m
or --memory
options for the docker run
command. The container memory contraint should be at least 350mb larger than the JAVA_HEAP_USAGE environment variable.
To restrict a container’s memory usage to 2g with Docker run:
docker run -d --name wavefront-proxy ... -e JAVA_HEAP_USAGE="1650m" -m 2g ...
To limit memory usage of the container in Kubernetes use the resources.limits.memory
property of a container definition. See the Kubernetes doc.
Customize Proxy Settings for Docker
When you run a Wavefront proxy inside a Docker container, you can tweak proxy configuration settings that are properties in the wavefront.conf
file directly from the Docker run
command. You use the WAVEFRONT_PROXY_ARGS environment variable and pass in the property name as a long form argument, preceded by --
.
For example, add -e WAVEFRONT_PROXY_ARGS="--pushRateLimit 1000"
to your docker run
command to specify a rate limit of 1000 pps for the proxy.
See the Wavefront Proxy configuration file for a full list.
Log Customization for Docker Containers
You can customize logging by mounting a customized log4j2.xml
file. Here’s an example for Docker:
--mount type=bind, src=<absolute_path>/log4j2.xml, dst=/etc/wavefront/wavefront-proxy/log4j2.xml
See Logging above for additional background.
Authenticate Incoming HTTP Requests at the Proxy
The Wavefront proxy can be configured to authenticate inbound traffic. Below are the steps for setting up the different supported authentication methods.
Authentication Method Static Token
Required properties: authStaticToken
In order for data sent via HTTP to the proxy to be accepted, the request will need to include a token matching whatever is configured for authStaticToken
. The token can be specified in the header or in the query string.
Step | Example |
---|---|
1. Set authMethod to STATIC_TOKEN . |
authMethod=STATIC_TOKEN |
2.
Set authStaticToken to a value that will be used as the authentication token. |
authStaticToken=token1234abcd |
3. Ensure that valid data sent to the proxy has the appropriate token included with the request. |
Authentication Method HTTP_Get or OAUTH2
Required properties: authTokenIntrospectionServiceUrl
Optional properties: authTokenIntrospectionAuthorizationHeader
, authResponseRefreshInterval
, authResponseMaxTtl
If you have a service that can validate tokens, use this approach.
- Use OAUTH2 if your Introspection Service is RFC7662 compliant.
- Otherwise, use HTTP_GET.
For either method, the service must return a 2xx code for valid tokens. According to the standard, OAUTH2-compliant services return a JSON object that contains the active field.
Step | Example |
---|---|
1. Set authMethod to HTTP_GET or OAUTH2 . |
authMethod=OAUTH2 |
2. Set authTokenIntrospectionServiceUrl to the appropriate token validation endpoint for your introspection service. Use {{token}} as the placeholder for the token. |
authTokenIntrospectionServiceUrl= |
3. If the token validation endpoint requires authentication, specify authTokenIntrospectionAuthorizationHeader . |
authTokenIntrospectionAuthorizationHeader= |
4.
Optionally, set authResponseRefreshInterval to specify how long to cache token validation results, in seconds, before re-authenticating against the introspection service. Default is 600 seconds (10 minutes). |
authResponseRefreshInterval=300
|
5. Optionally, set authResponseMaxTtl to specify the maximum amount of time, in seconds, to cache token validation results if the introspection service cannot be reached. Default is 86400 seconds (1 day). |
authResponseMaxTtl=21600
|
6. Ensure that valid data sent to the proxy has the appropriate token included with the request. |