Proxy files, logs, and configuration properties

Even without additional customization the Wavefront proxy ingests metrics and forwards them to the Wavefront service in a secure, fast, and reliable manner. If needed, you can customize your proxy.

  • Proxy configuration properties allow you to changing how the proxy processes your data. For example, you can change ports or perform other advanced installation management.
  • Proxy preprocessor rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.
  • Log files can help in case of problems.

Proxy File Paths

By default, proxy files are installed in the following locations.

  • Configuration properties: <wavefront_config_path>
    • Linux - /etc/wavefront/wavefront-proxy
    • Mac - /usr/local/etc/wavefront/wavefront-proxy
    • Windows - C:\Program Files (x86)\Wavefront\conf
  • Parse log data: <wavefront_log_path>
    • Linux - /var/log/wavefront
    • Mac - /usr/local/var/log/wavefront
    • Windows - C:\Program Files (x86)\Wavefront
    • For Docker containers, see logging info on docs.docker.com.

See Logging for details on log files and content.

  • <wavefront_spool_path>
    • Linux - /var/spool/wavefront-proxy
    • Mac - /usr/local/var/spool/wavefront-proxy
    • Windows - C:\Program Files (x86)\Wavefront\bin

Data Buffering

If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:

2013-11-18 18:02:35,061 WARN  [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]

By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:

2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared

Logging

The Wavefront proxy supports two log files: proxy log and blocked point log.

To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. Configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.

If you’re using proxies in containers, you can mount the log files, as discussed below.

Proxy Log

By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.

If you want to set logs for Jaeger and Zipkin integrations, see Logging for Jaeger and Zipkin.

Blocked Data Log

You can log all the raw blocked data separately or log different entities into their separate log files.

  • Log the block data separately
    Follow these steps:
    1. Open the <wavefront_config_path>/log4j2.xml configuration file.
    2. To log all the block data, uncomment the corresponding section. 
       <AsyncLogger name="RawBlockPoints" level="WARN" additivity="false">
    <AppenderRef ref="BlockedPointsFile" />
 </AsyncLogger>

      By default, blocked point entries are logged to the <wavefront_log_path>/wavefront-blocked-points.log file and the log file is rolled over every day when its size reaches 100MB. When there are 31 log files, older files are deleted. You can customize the configurations to suit your environment.

  • Set up separate log files for blocked entities
    Follow these steps:
    1. Uncomment or add the configurations under Appenders and Loggers in the <wavefront_config_path>/log4j2.xml configuration file. 
         <!-- Log the blocked histograms. If you don't need a separate log file for it,
   don't add this configuration to the file.-->
   <AsyncLogger name="RawBlockedHistograms" level="WARN" additivity="false">
      <AppenderRef ref="[Enter_Your_File_Name]"/>
  </AsyncLogger>

  <!-- Logs the blocked points for spans. If you don't need a separate log file for it,
  don't add this configuration to the file.-->
  <AsyncLogger name="RawBlockedSpans" level="WARN" additivity="false">
      <AppenderRef ref="[Enter_Your_File_Name]"/>
  </AsyncLogger>

  <AsyncLogger name="RawBlockPoints" level="WARN" ADDITIVITY="FALSE"/>
	   <AppenderRef ref=”BlockedPointsFile”/>
  </AsyncLogger>
    2. Add the names of the block points, which you uncommented in the log4j2.xml file, to the <wavefront_config_path>/wavefront.conf file.
      Example: 
         blockedPointsLoggerName = RawBlockedPoints

   # Add this if you added the appender for histograms in the log4j2.xml file.
   blockedHistogramsLoggerName = RawBlockedHistograms (RawBlockedPoints by default)

   # Add this if you added the appender for spans in the log4j2.xml file.
   blockedSpansLoggerName = RawBlockedSpans (RawBlockedPoints by default)

Configuration Properties

This section gives details on proxy configuration properties. All properties are also listed, in the wavefront.conf.default file on Github.

General Configuration Properties

Property Purpose Format and Example Since
agentMetricsPointTags Point tags and their values to be passed along with ~agent./ metrics.
Default: None.		 Comma-separated list of key-value pairs.
Ex: dc=west,env=prod		 3.24
block Regex pattern (java.util.regex) that input lines must match to be filtered out. Input lines are checked against the pattern as they come in and before the prefix is prepended. Renamed from blackListRegex to block in proxy 9.x.

Use preprocessor rules for finer-grained control.		 Valid regex pattern.
Ex: Filter out points that begin with qa., development., or test.: ^(qa|development|test).		 3.1/9.x
blockedPointsLoggerName Controls how the blocked points are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints. 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
blockedHistogramsLoggerName Controls how the blocked histograms are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints. 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
blockedSpansLoggerName Controls how the blocked spans are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints. 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
buffer Location of buffer files for saving failed transmissions for retry. Valid path on the local file system.
Ex: <wf_spool_path>/buffer		 The start
customSourceTags Point tag keys to use as 'source' if no 'source' or 'host' field is present.
Default: fqdn, hostname.		 Comma-separated list of point tag keys.
Ex: fqdn, hostname		 3.14
dataPrefillCutoffHours Defines the cut-off point for what is considered a valid timestamp for pre-dated points. Default (and recommended) value is 24 (1 day), so all the data points from more than 1 day in future are rejected. Positive integer.
Ex: 12		 4.1
dataBackfillCutoffHours The cut-off point for what is considered a valid timestamp for back-dated points. We do not recommend setting this value larger than 1 year unless backfilling or migrating historic data.
Default: 8760 (1 year), so all points older than 1 year are rejected.		 Positive integer.
Ex: 8760		 4.1
deltaCountersAggregationListenerPorts Port to listen to the Wavefront-formatted delta counter data. Other data formats are rejected at this port. Pre-aggregating delta counters at the proxy, helps reduce the outbound point rate. Use this property in conjunction with deltaCountersAggregationIntervalSeconds to limit the number of points per second for delta counters.
Default: none.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 12878
Ex: 12878,12879		 6.0
deltaCountersAggregationIntervalSeconds Interval between flushing aggregating delta counters to Wavefront. Use this property in conjunction with deltaCountersAggregationListenerPorts to send points to the port(s) in batches, thereby limiting the number of points per second.
Default: 30 seconds. 		Number of seconds.
Ex: 45		 6.0
fileBeatPort TCP port to listen on for Filebeat data.
Default: none.		 A port number.
Ex: 5044 		4.1
flushThreads Number of threads that flush data to the server. Setting this value too high results in sending batches that are too small to the Wavefront server and wasting connections. Values between 6 and 16 are a good starting point. This setting is per listening port.
Default: The number of available processors (min 4).		 Positive integer.
Ex: 16		 3.14
graphiteDelimiters Characters that should be replaced by dots, in case they were escaped within Graphite and collectd before sending. A common delimiter is the underscore character; so if you extract a hostname field with the value web04_www, it is changed to web04.www. A concatenation of delimiter characters, without any separators.  
graphiteFormat Indexes of fields within Graphite and collectd metric names that correspond to a hostname. For example, if your metrics have the format: collectd.prod.www04.cpu.loadavg.1m, specify the 3rd and 2nd indexes (www04.prod) to be extracted and treated as the hostname. The remainder collectd.cpu.loadavg.1m is treated as the metric name. Comma-separated list of indexes.
Ex: 4, 2, 5 Ex: 3 		 
graphitePorts TCP ports to listen on for Graphite data. Define which of the segments in your Graphite metrics map to a hostname in the graphiteFormat property. Default: None. Comma-separated list of available port numbers. Can be a single port.
Ex: 2003
Ex: 2003, 2004 		 
gzipCompression If set to true, metric traffic from the proxy to the Wavefront endpoint is gzip-compressed.
Default: true.		 true or false
Ex: true		  
gzipCompressionLevel Sets the gzip compression level if gzipCompression is enabled. The level vary from, 1 to 9. Higher compression levels slightly reduces the volume of traffic between the proxy and Wavefront, but uses more CPU.
Default: 4.		 Positive integer ranging from 1 to 9.
Ex: 4		 6.0
Histogram configuration properties Properties specific to histogram distributions, listed in a separate table below. 4.31
hostname A name unique across your account representing the machine that the proxy is running on. The hostname is not used to tag your metrics; rather, it's used to tag proxy metrics, such as JVM statistics, per-proxy point rates, and so on. A string containing alphanumeric characters and periods.  
httpConnectTimeout HTTP connect timeout (in milliseconds).
Default: 5000 (5s).		 Positive integer.
Ex: 5000
4.1
httpRequestTimeout HTTP request timeout (in milliseconds). We do not recommend setting this value to be higher than 20000. Recommended value for most configurations is 10000 (10 seconds).
Default: 10000 (10s).		 Positive integer.
Ex: 10000
4.1
httpMaxConnTotal Max number of total connections to keep open.
Default: 200		 Positive integer.
Ex: 100
httpMaxConnPerRoute Max number of connections per route to keep open.
Default: 100.		 Positive integer.
Ex: 50
httpUserAgent Override User-Agent in request headers. Can help bypass excessively restrictive filters on the HTTP proxy. Default user agent: Wavefront-Proxy/<version>. A string.
Ex: 'Mozilla/5.0'
4.1
idFile Location of the PID file for the wavefront-proxy process.
Default: <dshell>/.id		 Valid path on the local file system. This option is ignored when ephemeral=true.  
jsonListenerPorts TCP ports to listen on for incoming JSON-formatted metrics.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.  
listenerIdleConnectionTimeout Close idle inbound connections after specified time in seconds.
Default: 300 		Number of seconds. 4.31
logsIngestionConfigFile The file containing instructions for parsing log data into metrics. See Log Data Metrics Integration. Default: <cfg_path>/logsIngestion.yaml. Valid path on the local file system. 4.1
opentsdbPorts TCP ports to listen on for incoming OpenTSDB-formatted data.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 4242
3.1
picklePorts TCP ports to listen on for incoming data in Graphite pickle format (from carbon-relay).
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 5878
3.20
prefix String to prepend before every metric name. For example, if you set prefix to 'production', a metric that is sent to the proxy as cpu.loadavg.1m is sent from the proxy to Wavefront as production.cpu.loadavg.1m. You can include longer prefixes such as production.nyc.dc1.
Default: None.		 A lowercase alphanumeric string, with periods separating segments. You do not need to include a trailing period.
Ex: production
Ex: production.nyc.dc1
  
preprocessorConfigFile Path to the optional preprocessor config file containing preprocessor rules for filtering and rewriting metrics.
Default: None.		 Valid path on the local file system.
Ex: <cfg_path>/rules.yaml
 4.1
privateKeyPath Path to PKCS#8 private key file in PEM format. Incoming TLS/SSL connections access this private key. 8.0
privateCertPath Path to X.509 certificate chain file in PEM format. Incoming TLS/SSL connections access this certificate. 8.0
proxyHost HTTP proxy host to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyPort. A string.
Ex: proxy.local
 3.23
proxyPassword When used with proxyUser, sets credentials to use with the HTTP proxy if the proxy requires authentication. A string.
Ex: validPassword123
3.23
proxyPort HTTP proxy port to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyHost. A port number.
Ex: 8080
3.23
proxyUser When used with proxyPassword, sets credentials to use with the HTTP proxy if the proxy requires authentication. A string.
Ex: validUser
3.23
pushBlockedSamples Number of blocked points to print to the log immediately following each summary line (every 10 flushes). If 0, print none. If you see a non-zero number of blocked points in the summary lines and want to debug what that data is, set this property to 5.
Default: 5 		0 or a positive integer.
Ex: 5
 
pushFlushInterval Milliseconds to wait between each flush to Wavefront.
Default: 1000.		 An integer equal to or greater than 1000.
Ex: 1000
 
pushFlushMaxPoints Maximum number of points to send to Wavefront during each flush.
Default: 40,000.		 Positive integer.
Ex: 40000
 
pushFlushMaxHistograms Maximum number of histograms to send to Wavefront during each flush.
Default: 10,000.		 Positive integer.
Ex: 10000
6.0
pushFlushMaxSpans Maximum number of spans to send to Wavefront during each flush.
Default: 5,000.		 Positive integer.
Ex: 5000
6.0
pushFlushMaxSpanLogs Maximum number of span logs to send to Wavefront during each flush.
Default: 1,000.		 Positive integer.
Ex: 1000
6.0
pushListenerHttpBufferSize Maximum allowed request size (in bytes) for incoming HTTP requests on Wavefront, OpenTSDB, or Graphite ports.
Default: 16777216 (16MB). 		Ex: 8388608 4.31
pushListenerMaxReceivedLength Maximum line length for received points in plaintext format on Wavefront, OpenTSDB, or Graphite ports.
Default: 32KB		 Positive integer.
Ex: 4096
4.31
pushListenerPorts Port to listen on for incoming data. A single port definition can accept both HTTP and TCP data. For HTTP data, make a POST to this proxy port with an empty header, and the line terminated data format. If you want to use HTTPS/TLS, set the tlsPort, privateKeyPath, and privateCertPath as well.
Default: 2878.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 2878
Ex: 2878,2879,2880
  
pushLogLevel Frequency to print status information on the data flow to the log. SUMMARY prints a line every 60 flushes, while DETAILED prints a line on each flush. None, SUMMARY, or DETAILED
Ex: SUMMARY
Before 6.0.
pushMemoryBufferLimit Maximum number of points that can stay in memory buffers before spooling to disk. Setting this value lower than default reduces memory usage but forces the proxy to queue points by spooling to disk more frequently, if you have points arriving at the proxy in short bursts. Default: 16 * pushFlushMaxPoints. Minimum: pushFlushMaxPoints. Positive integer.
Ex: 640000
 4.1
pushRateLimit Maximum number of points per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 20000
 4.1
pushRateLimitHistograms Maximum number of histograms per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 20000
 6.0
pushRateLimitSpans Maximum number of spans per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 10000
 6.0
pushRateLimitSpanLogs Maximum number of span logs per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 10000
 6.0
pushRateLimitMaxBurstSeconds Max number of burst seconds to allow when rate limiting to smooth out uneven traffic. Set to 1 when doing data backfills.
Default: 10		 Positive integer.
Ex: 20
pushRelayListenerPorts Ports to receive the data sent to the relay. In environments where direct outbound connections to Wavefront servers are not possible, you can use another Wavefront proxy that has outbound access to act as a relay and forward all the data received on that endpoint (from direct data ingestion clients and/or other proxies) to Wavefront servers.
Default: none.		 Comma-separated list of available port numbers. Can be a single port. Ex: 2978
Ex: 2978,2979		 6.0
pushRelayHistogramAggregator If set to true, aggregates the histogram distributions received on the relay port.
Default: false.		 true or false
Ex: true
 6.0
pushValidationLevel Level of validation to perform on incoming data before sending the data to Wavefront. If NO_VALIDATION, all data is sent forward. If NUMERIC_ONLY, data is checked to make sure that it is numerical and dropped locally if it is not. NUMERIC_ONLY or NO_VALIDATION
Ex: NUMERIC_ONLY
 
rawLogsMaxReceivedLength Maximum line length for received raw logs.
Default: 4096. 		Positive integer.
Ex: 4096
4.31
rawLogsPort TCP port to listen on for log data.
Default: none.		 A port number.
Ex: 5045
4.4
retryBackoffBaseSeconds For exponential back-off when retry threads are throttled, the base (a in a^b) in seconds.
Default: 2.0.		 Positive number, integer or decimal.
Ex: 2.0
 
retryThreads Number of threads retrying failed transmissions. If no value is specified, defaults to the number of processor cores available to the host or 4, whichever is greater. Every retry thread uses a separate buffer file (capped at 2GB) to persist queued data points, so the number of threads controls the maximum amount of space that the proxy can use to buffer points locally. Positive integer.
Ex: 4
Before 6.0.
server The API URL of the Wavefront server in the format https://<wf_instance>.wavefront.com/api/.    
soLingerTime Enable SO_LINGER with the specified linger time in seconds. Set this value to 0 when running in a high-availability configuration under a load balancer.
Default: -1 (disabled).
0 or a positive integer.
Ex: 0 		4.1
splitPushWhenRateLimited Whether to split the push batch size when the push is rejected by Wavefront due to rate limit.
Default: false.		 true or false
Ex: false
 
tlsPorts Comma-separated list of ports to be used for incoming TLS/SSL connections. To set up a port to use TLS/SSL, you specify pushListenerPorts, tlsPorts, privateKeyPath, and privateCertPath. 8.0
allow Regex pattern (java.util.regex). Input lines are checked against the pattern as they come in and before the prefix is prepended. Only input lines that match are accepted. Renamed from whiteListRegex to allow in proxy 9.x.

Use preprocessor rules for finer-grained control. 		Valid regex pattern.
Ex: ^(production|stage).
Allows points that begin with production. and stage.
3.1
writeHttpJsonListenerPorts Ports to listen on for incoming data from the collected write_http plugin.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 4878
3.14
rawLogsHttpBufferSize The maximum request size (in bytes) for incoming HTTP requests with tracing data.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216 		4.38
trafficShaping Enables intelligent traffic shaping based on the data receive rate over the last 5 minutes.
Default: false		 true or false 9.0
trafficShapingQuantile Sets the quantile for traffic shaping.
Default: 75		 An integer
Ex: 99
The 99th percentile of the received rate in the last 5 minutes, will be used as a basis for the rate limiter. 		9.0
trafficShapingHeadroom Sets the headroom multiplier for traffic shaping when there's backlog.
Default: 1.15 (15% headroom) 		Number from 1.0 to 1.99
Ex: 1.05 (5% headroom) 		9.0
sqsBuffer Use AWS SQS for buffering transmissions.
Default: false		 true or false 9.0
sqsQueueNameTemplate The replacement pattern for naming the SQS queues. Ex: wf-proxy-{{id}}-{{entity}}-{{port}} results in a queue named wf-proxy-id-points-2878 9.0
sqsQueueIdentifier An identifier for identifying the proxies in SQS. A string
Ex: wavefront 		9.0
sqsQueueRegion The AWS Region name the queue lives in. A string
Ex: us-west-2 		9.0

Authentication Proxy Properties

Because the proxy is running in your local network by default, communication to the proxy is un-authenticated. If you want to authenticate inbound traffic to the proxy, use the settings in this section.

The Wavefront proxy must authenticate outbound traffic to the Wavefront service. See Authenticate Incoming HTTP Requests at the Proxy for step-by-step instructions.

Property Purpose Format and Example
authMethod Authentication is done by a token. Specify the tokens either in the query string ("token=" and "api_key=" parameters) or in the request headers ("X-AUTH-TOKEN: ", "Authorization: Bearer", "Authorization: " headers). TCP streams are disabled when authentication is turned on.
Default: NONE.		 Allowed authMethod: NONE, STATIC_TOKEN, HTTP_GET, OAUTH2.
Ex: OAUTH2
authTokenIntrospectionServiceUrl URL for the token introspection endpoint used to validate tokens for incoming HTTP requests (for example, authTokenIntrospectionServiceUrl = https://auth.acme.de/api/token/<token>/validate). Required when authMethod is OAUTH2 or HTTP_GET
Default: None.		 Valid URL
authTokenIntrospectionAuthorizationHeader Optional credentials for use with the token introspection endpoint if the endpoint requires authentication. For example:
authTokenIntrospectionAuthorizationHeader= Authorization: Bearer <token>
authResponseRefreshInterval Cache TTL (in seconds) for token validation results (re-authenticate when expired).
Default: 600 seconds.		 Integer
Ex: 1200
authResponseMaxTtl Maximum allowed cache TTL (in seconds) for token validation results when the token introspection service is unavailable.
Default: 86400 seconds (1 day).		 Integer
Ex: 3600
authStaticToken Static token that is considered valid for all incoming HTTP requests.
Required when authMethod = STATIC_TOKEN. For example, authStaticToken=token1234abcd
Default: None 		Valid token

Tracing Proxy Properties

Property Purpose Format /Example
traceAlwaysSampleErrors Always sample spans with an error tag ignoring other sampling configuration.
Default: true		 true or false
Ex: false
traceJaegerHttpListenerPorts TCP ports to receive Jaeger Thrift formatted data via HTTP. The data is then sent to Wavefront in Wavefront span format.
Default: None.
Version: Since 6.0		 Comma-separated list of available port numbers. Can be a single port.
traceJaegerListenerPorts TCP ports to receive Jaeger Thrift formatted data via TChannel. The data is then sent to Wavefront in Wavefront span format.
Default: None. 		Comma-separated list of available port numbers. Can be a single port.
traceJaegerGrpcListenerPorts Ports to receive Jaeger Protobuf formatted data over gRPC.
Default: None.
Version: Since 9.0		 Comma-separated list of available port numbers. Can be a single port.
customTracingListenerPorts TCP ports to receive spans and derive RED metrics from the SDKs that send raw data to Wavefront.
Default: None.
Version: Since 6.0 		Comma-separated list of available port numbers. Can be a single port.
traceListenerMaxReceivedLength Maximum line length for received spans and span logs.
Default: 1MB		 Integer
Ex: 1048576
traceListenerPorts TCP ports that listen to incoming spans from the Wavefront SDKs that collect trace data, collect metrics and histograms, and Wavefront SDKs that instrument frameworks.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 30000
Ex: 30000, 30001
traceSamplingDuration Minimum duration of the tracing spans that can be sent to Wavefront for trace data sampling.
Default: 0 (send all generated spans). 		Number of milliseconds.
Ex: 45
traceSamplingRate Percentage of all generated spans to send to Wavefront for trace data sampling.
Default: 1.0 (send all generated spans). 		Number from 0.0 to 1.0.
Ex: .1
traceZipkinListenerPorts TCP ports to listen on for Zipkin formatted data. Recommended: The default Zipkin Collector port (9411).
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
customTracingApplicationName Custom application name for spans received on the customTracingListenerPorts that don't have the application tag.
Default: defaultApp.
Version: Since 9.0		 customTracingApplicationName=MyApplication
customTracingServiceName Custom service name for spans received on the customTracingListenerPorts that don't have the service tag.
Default: defaultService.
Version: Since 9.0		 customTracingServiceName=MyService
traceJaegerApplicationName Custom application name for traces received on Jaeger's traceJaegerListenerPorts or traceJaegerHttpListenerPorts. traceJaegerApplicationName=MyJaegerDemo
traceZipkinApplicationName Custom application name for traces received on Zipkin's traceZipkinListenerPorts. traceZipkinApplicationName=MyZipkinDemo
traceDerivedCustomTagKeys Comma separated list of custom tag keys to include as metric tags for the derived RED (Request, Error, Duration) metrics. Applicable to Jaeger and Zipkin integration only. traceDerivedCustomTagKeys=tenant, env, location
traceListenerHttpBufferSize The maximum request size (in bytes) for incoming HTTP requests with tracing data.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216

Histogram Configuration Properties

Wavefront supports additional histogram configuration properties, shown in the following table. Note the requirements on the state directory and the effect of the two persist properties listed at the bottom of the table.

PropertyDescriptionFormat
histogramAccumulatorResolveInterval Interval in milliseconds to write back accumulation changes from memory cache to disk. Only applicable when memory cache is enabled. Increasing this setting reduces storage IO pressure but might increase heap memory use. Default: 100. Positive integer.
histogramAccumulatorFlushInterval Interval in milliseconds to check for histograms that need to be sent to Wavefront according to their histogramMinuteFlushSecs settings. Default: 1000. Positive integer.
histogramAccumulatorFlushMaxBatchSize Max number of histograms to move to the outbound queue in one flush. Default: no limit. Positive integer.
histogramDayAccumulators Number of accumulators per day port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. Positive integer.
histogramDayAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramDayAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramDayAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramDayCompression * 7 Positive integer.
histogramDayAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramDayCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval [20;1000].
histogramDayFlushSecs Time-to-live, in seconds, for a day granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 18000 (5 hours). Positive integer.
histogramDayListenerPorts TCP ports to listen on for histograms to be aggregated by day. Default: 40003. Comma-separated list of ports.
histogramDayMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. Boolean.
histogramDistAccumulators Number of accumulators per distribution port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: number of available CPU cores. Positive integer.
histogramDistAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramDistAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramDistAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramDistCompression * 7 Positive integer.
histogramDistCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval [20;1000].
histogramDistFlushSecs Number of seconds to keep a new distribution bin open for new samples, before the intermediary is sent to Wavefront. Default: 70. Positive integer.
histogramDistListenerPorts TCP ports to listen on for ingesting histogram distributions. Default: 40000. Comma-separated list of ports. Can be a single port.
histogramDistMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (Aggregating more than 1 distribution per second per time series). Default: false. Boolean.
histogramDistAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramHttpBufferSize Since 4.40. The maximum request size (in bytes) for incoming HTTP requests on histogram ports.
Default: 16MB.		 Buffer size in bytes.
Ex: 16777216
histogramHourAccumulators Number of accumulators per hour port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. Positive integer.
histogramHourAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramHourAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramHourAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramMinuteCompression * 7 Positive integer.
histogramHourAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramHourCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval [20;1000].
histogramHourFlushSecs Time-to-live, in seconds, for an hour granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 4200. Positive integer.
histogramHourListenerPorts TCP ports to listen on for histograms to be aggregated by hour. Default: 40002. Comma-separated list of ports. Can be a single port.
histogramHourMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. Boolean.
histogramMaxReceivedLength Maximum line length for received histogram points. Default: 65536. Positive integer.
histogramMinuteAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramMinuteListenerPorts TCP ports to listen on for histograms to be aggregated by minute. Default: 40001. Comma-separated list of ports. Can be a single port.
histogramMinuteAccumulators Number of accumulators per minute port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. Positive integer.
histogramMinuteAccumulatorSize Expected upper bound of concurrent accumulations. Should be approximately the number of timeseries * 2 (use a higher multiplier if out-of-order points more than 1 minute apart are expected). Setting this value too high results in excessive disk space usage, setting this value too low may cause severe performance issues. Default: 100000. Positive integer.
histogramMinuteAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramMinuteCompression A bound on the number of centroids per histogram. Default: 32. Positive integer in the interval. [20;1000].
histogramMinuteFlushSecs Time-to-live, in seconds, for a minute granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 70. Positive integer.
histogramMinuteMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. Boolean.
histogramMinuteAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramMinuteCompression * 7 Positive integer.
histogramProcessingQueueScanInterval Interval in milliseconds between checks for new entries in the processing queue. Default: 20. Positive integer.
histogramReceiveBufferFlushInterval Sets maximum time in milliseconds that incoming points can stay in the receive buffer when incoming traffic volume is very low. Default: 100. Positive integer.
histogramStateDirectory Directory for persistent proxy state, must be writable. Before being flushed to Wavefront, histogram data is persisted on the filesystem where the Wavefront proxy resides. If the files are corrupted or the files in the directory can't be accessed, the proxy reports the problem in its log and fails back to using in-memory structures. In this mode, samples can be lost if the proxy terminates without draining its queues. Default: /var/spool/wavefront-proxy. A valid path on the local file system.
persistAccumulator Whether to persist accumulation state. We suggest keeping this setting enabled unless you are not using hour and day level aggregation and consider losing up to 1 minute worth of data during proxy restarts acceptable. Default: true. .
persistMessages Whether to persist received metrics to disk. Default: true. Boolean.
pushRelayHistogramAggregatorAccumulatorSize Since 6.0. Max number of concurrent histogram to accumulations at the relay ports. The value is approximately the number of time series * 2 (use a higher multiplier if out-of-order points more than 1 bin apart are expected). Setting this value too high will cause excessive disk space usage, setting this value too low may cause severe performance issues. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 32.		 Positive integer.
pushRelayHistogramAggregatorCompression Since 6.0. Number of centroids per histogram. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 32.		 Must be between 20 and 1000.
Ex: 40
pushRelayHistogramAggregatorFlushSecs Since 6.0. Interval in milliseconds to check for histograms that have accumulated at the relay ports before sending data to Wavefront. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 70.		 Number of milliseconds.
Ex: 80

Configure a Proxy in a Docker Container

You can configure a proxy in a Docker container by running one of the following commands:

dockerhub:

docker run -d -e WAVEFRONT_URL=https://<myinstance>.wavefront.com/api/ -e
WAVEFRONT_TOKEN=YOUR_API_TOKEN -p 2878:2878 wavefronthq/proxy:latest

Harbor:

docker run -d -e WAVEFRONT_URL=https://<myinstance>.wavefront.com/api/ -e
WAVEFRONT_TOKEN=YOUR_API_TOKEN -p 2878:2878 projects.registry.vmware.com/tanzu_observability/proxy:latest

For both commands:

  • WAVEFRONT_TOKEN is the API token for the account
  • <myinstance> is the URL of the Wavefront instance, for example, https://example.wavefront.com
  • The proxy uses port 2878 by default. The proxy configuration file supports setting explicit ports for different kinds of metrics.

Proxy Versions for Containers

For containers, the proxy image version is determined by the image property in the configuration file. You have these choices:

  • Use wavefronthq/proxy:latest for an image on dockerhub.
  • Use projects.registry.vmware.com/tanzu_observability/proxy:latest for an image on Harbor.
  • Specify a proxy version explicitly in the proxy configuration file.

The proxies are not stateful. Your configuration is managed in your yaml file. It’s safe to use proxy:latest – we ensure that proxies are backward compatible.

Restrict Memory Usage for a Docker Container

To restrict memory usage of the container using Docker, you need to add a JAVA_HEAP_USAGE environment variable and restrict memory using the -m or --memory options for the docker run command. The container memory contraint should be at least 350mb larger than the JAVA_HEAP_USAGE environment variable.

To restrict a container’s memory usage to 2g with Docker run:

docker run -d --name wavefront-proxy ... -e JAVA_HEAP_USAGE="1650m" -m 2g ...

To limit memory usage of the container in Kubernetes use the resources.limits.memory property of a container definition. See the Kubernetes doc.

Customize Proxy Settings for Docker

When you run a Wavefront proxy inside a Docker container, you can tweak proxy configuration settings that are properties in the wavefront.conf file directly from the Docker run command. You use the WAVEFRONT_PROXY_ARGS environment variable and pass in the property name as a long form argument, preceded by --.

For example, add -e WAVEFRONT_PROXY_ARGS="--pushRateLimit 1000" to your docker run command to specify a rate limit of 1000 pps for the proxy.

See the Wavefront Proxy configuration file for a full list.

Log Customization for Docker Containers

You can customize logging by mounting a customized log4j2.xml file. Here’s an example for Docker:

--mount type=bind, src=<absolute_path>/log4j2.xml, dst=/etc/wavefront/wavefront-proxy/log4j2.xml

See Logging above for additional background.

Authenticate Incoming HTTP Requests at the Proxy

The Wavefront proxy can be configured to authenticate inbound traffic. Below are the steps for setting up the different supported authentication methods.

Authentication Method Static Token


Required properties: authStaticToken


In order for data sent via HTTP to the proxy to be accepted, the request will need to include a token matching whatever is configured for authStaticToken. The token can be specified in the header or in the query string.

StepExample
1. Set authMethod to STATIC_TOKEN. authMethod=STATIC_TOKEN
2. 
Set authStaticToken to a value that will be used as the authentication token. authStaticToken=token1234abcd
3. 
Ensure that valid data sent to the proxy has the appropriate token included with the request.  

Authentication Method HTTP_Get or OAUTH2



Required properties: authTokenIntrospectionServiceUrl


Optional properties: authTokenIntrospectionAuthorizationHeader, authResponseRefreshInterval, authResponseMaxTtl


If you have a service that can validate tokens, use this approach.

  • Use OAUTH2 if your Introspection Service is RFC7662 compliant.
  • Otherwise, use HTTP_GET.

For either method, the service must return a 2xx code for valid tokens. According to the standard, OAUTH2-compliant services return a JSON object that contains the active field.

StepExample
1. Set authMethod to HTTP_GET or OAUTH2. authMethod=OAUTH2
2. Set authTokenIntrospectionServiceUrl to the appropriate token validation endpoint for your introspection service. Use {{token}} as the placeholder for the token. authTokenIntrospectionServiceUrl=
https://auth.acme.corp/api/token/{{token/validate}}
3. If the token validation endpoint requires authentication, specify authTokenIntrospectionAuthorizationHeader. authTokenIntrospectionAuthorizationHeader=
Authorization: Bearer token123xyz

4. 
Optionally, set authResponseRefreshInterval to specify how long to cache token validation results, in seconds, before re-authenticating against the introspection service. Default is 600 seconds (10 minutes). authResponseRefreshInterval=300

5. Optionally, set authResponseMaxTtl to specify the maximum amount of time, in seconds, to cache token validation results if the introspection service cannot be reached. Default is 86400 seconds (1 day). authResponseMaxTtl=21600


6. 
Ensure that valid data sent to the proxy has the appropriate token included with the request.  

Troubleshooting

TOKEN EXPIRED Error

  • Symptom: When you select Browse > Proxies, you see a TOKEN EXPIRED error.
  • Cause: This error usually means that the account that owned the token was disabled, for example, because a user left a company. Tokens do not actually expire. We’re working on improving the message.
  • Resolution: Create a new token for those metrics and update the proxy config file to use that new token.

KB Articles

See the following KB articles for additional proxy troubleshooting help.

proxies