Proxy files, logs, and configuration properties

Even without additional customization, the Wavefront proxy ingests metrics and forwards them to VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront) in a secure, fast, and reliable manner. If needed, you can customize your proxy.

  • Proxy configuration properties allow you to change how the proxy processes your data. For example, you can change ports or perform other advanced installation management.

  • Proxy preprocessor rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.
  • Log files can help in case of problems.

Proxy File Paths

By default, proxy files are installed in the following locations.

  • Configuration properties: <wavefront_config_path>
    • Linux - /etc/wavefront/wavefront-proxy
    • Mac - /usr/local/etc/wavefront/wavefront-proxy
    • Windows - C:\Program Files (x86)\Wavefront\conf
  • Parse log data: <wavefront_log_path>
    • Linux - /var/log/wavefront
    • Mac - /usr/local/var/log/wavefront
    • Windows - C:\Program Files (x86)\Wavefront
    • For Docker containers, see logging info on docs.docker.com.

    See Logging for details on log files and content.

  • <wavefront_spool_path>
    • Linux - /var/spool/wavefront-proxy
    • Mac - /usr/local/var/spool/wavefront-proxy
    • Windows - C:\Program Files (x86)\Wavefront\bin

Data Buffering

If the Wavefront proxy is unable to post received data to the Operations for Applications servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Operations for Applications servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:

2013-11-18 18:02:35,061 WARN  [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]

By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Operations for Applications servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:

2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared

Logging

The Wavefront proxy supports two log files: proxy log and blocked point log.

To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. Configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.

If you’re using proxies in containers, you can mount the log files, as discussed below.

Proxy Log

By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.

If you want to set logs for Jaeger and Zipkin integrations, see Logging for Jaeger and Zipkin.

Blocked Data Log

You can log all the raw blocked data separately or log different entities into their separate log files.

  • Log the block data separately
    Follow these steps:
    1. Open the <wavefront_config_path>/log4j2.xml configuration file.
    2. To log all the block data, uncomment the corresponding section. 
       <AsyncLogger name="RawBlockPoints" level="WARN" additivity="false">
    <AppenderRef ref="BlockedPointsFile" />
 </AsyncLogger>

      By default, blocked point entries are logged to the <wavefront_log_path>/wavefront-blocked-points.log file and the log file is rolled over every day when its size reaches 100MB. When there are 31 log files, older files are deleted. You can customize the configurations to suit your environment.

  • Set up separate log files for blocked entities
    Follow these steps:
    1. Uncomment or add the configurations under Appenders and Loggers in the <wavefront_config_path>/log4j2.xml configuration file. 
         <!-- Log the blocked histograms. If you don't need a separate log file for it,
   don't add this configuration to the file.-->
   <AsyncLogger name="RawBlockedHistograms" level="WARN" additivity="false">
      <AppenderRef ref="[Enter_Your_File_Name]"/>
  </AsyncLogger>

  <!-- Logs the blocked points for spans. If you don't need a separate log file for it,
  don't add this configuration to the file.-->
  <AsyncLogger name="RawBlockedSpans" level="WARN" additivity="false">
      <AppenderRef ref="[Enter_Your_File_Name]"/>
  </AsyncLogger>

  <AsyncLogger name="RawBlockPoints" level="WARN" ADDITIVITY="FALSE"/>
	   <AppenderRef ref=”BlockedPointsFile”/>
  </AsyncLogger>
    2. Add the names of the block points, which you uncommented in the log4j2.xml file, to the <wavefront_config_path>/wavefront.conf file.
      Example: 
         blockedPointsLoggerName = RawBlockedPoints

   # Add this if you added the appender for histograms in the log4j2.xml file.
   blockedHistogramsLoggerName = RawBlockedHistograms (RawBlockedPoints by default)

   # Add this if you added the appender for spans in the log4j2.xml file.
   blockedSpansLoggerName = RawBlockedSpans (RawBlockedPoints by default)
     click for top of page

Configuration Properties

This section gives details on the proxy configuration properties. All properties are also listed, in the wavefront.conf.default file on GitHub.

General Configuration Properties

Property Purpose Format and Example Since
agentMetricsPointTags Point tags and their values to be passed along with ~agent.* metrics.
Default: none		 Comma-separated list of key-value pairs.
Ex: dc=west,env=prod		 3.24
block
(Renamed from blackListRegex in proxy 9.x.)		 Regex pattern (java.util.regex) that input lines must match to be filtered out. Input lines are checked against the pattern as they come in and before the prefix is prepended.

Use preprocessor rules for finer-grained control.

 Valid regex pattern.
Ex: ^(qa|development|test)., which filters out all points that begin with qa., development., or test.		 3.1/9.x
blockedPointsLoggerName Controls how the blocked points are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
blockedHistogramsLoggerName Controls how the blocked histograms are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
blockedSpansLoggerName Controls how the blocked spans are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
buffer Location of buffer files for saving failed transmissions for retry. Valid path on the local file system.
Ex: <wf_spool_path>/buffer		 Initial
customSourceTags Point tag keys to use as source if no source or host field is present.
Default: fqdn,hostname		 Comma-separated list of point tag keys.
Ex: fqdn,hostname		 3.14
dataPrefillCutoffHours Defines the cut-off point for what is considered a valid timestamp for pre-dated points.
Default (and recommended): 24, so all the data points from more than 1 day in future are rejected.		 Number of hours.
Ex: 12		 4.1
dataBackfillCutoffHours The cut-off point for what is considered a valid timestamp for back-dated points. We do not recommend setting this value larger than 1 year unless backfilling or migrating historic data.
Default (and recommended): 8760, so all points older than 1 year are rejected.		 Number of hours.
Ex: 8760		 4.1
deltaCountersAggregationListenerPorts Ports to listen for delta counter data. Other data formats are rejected at this port. Pre-aggregating delta counters at the proxy, helps reduce the outbound point rate. Use this property in conjunction with deltaCountersAggregationIntervalSeconds to limit the number of points per second for delta counters.
Default: none		 Comma-separated list of available port numbers or a single port number.
Ex: 12878,12879
Ex: 12878		 6.0
deltaCountersAggregationIntervalSeconds Interval between flushing aggregating delta counters to the Operations for Applications service. Use this property in conjunction with deltaCountersAggregationListenerPorts to send points to the port(s) in batches, thereby limiting the number of points per second.
Default: 30 		Number of seconds.
Ex: 45		 6.0
ephemeral If set tofalse, you must also set the idFile parameter.
Default: true 		Boolean.
Ex: false
fileBeatPort TCP port to listen on for Filebeat data.
Default: none		 A port number.
Ex: 5044 		4.1
flushThreads Number of threads that flush data to the server. Setting this value too high results in sending batches that are too small to the Operations for Applications service and wasting connections. Values between 6 and 16 are a good starting point. This setting is per listening port.
Default: The number of available processors (min 4).		 Positive integer.
Ex: 16		 3.14
graphiteDelimiters Characters that should be replaced by dots, in case they were escaped within Graphite and collectd before sending. A common delimiter is the underscore character, so if you extract a hostname field with the value web04_www, it is changed to web04.www. A concatenation of delimiter characters, without any separators.  
graphiteFormat Indexes of fields within Graphite and collectd metric names that correspond to a hostname. For example, if your metrics have the format: collectd.prod.www04.cpu.loadavg.1m, specify the 3rd and 2nd indexes (www04.prod) to be extracted and treated as the hostname. The remainder (collectd.cpu.loadavg.1m) is treated as the metric name. Comma-separated list of indexes.
Ex: 4,2,5
Ex: 3 		 
graphitePorts TCP ports to listen on for Graphite data. Define which of the segments in your Graphite metrics map to a hostname in the graphiteFormat property.
Default: none		 Comma-separated list of available port numbers. Can be a single port.
Ex: 2003
Ex: 2003,2004 		 
gzipCompression If set to true, metric traffic from the proxy to the Operations for Applications endpoint is gzip-compressed.
Default: true		 Boolean.
Ex: true		  
gzipCompressionLevel Sets the gzip compression level if gzipCompression is enabled. Higher compression levels slightly reduces the volume of traffic between the proxy and the Operations for Applications service, but uses more CPU.
Default: 4		 A value from 1 to 9.
Ex: 4		 6.0
Histogram configuration properties Properties specific to histogram distributions, listed in a separate table below. 4.31
hostname A name unique across your account representing the machine that the proxy is running on. The hostname is not used to tag your metrics; rather, it's used to tag proxy metrics, such as JVM statistics, per-proxy point rates, and so on. A string containing alphanumeric characters and periods.  
httpConnectTimeout HTTP connect timeout (in milliseconds).
Default: 5000 (5s)		 Positive integer.
Ex: 5000
4.1
httpRequestTimeout HTTP request timeout (in milliseconds). We do not recommend setting this value to be higher than 20000. Recommended value for most configurations is 10000 (10 seconds).
Default: 10000 (10s)		 Positive integer.
Ex: 10000
4.1
httpMaxConnTotal Max number of total connections to keep open.
Default: 200		 Positive integer.
Ex: 100
httpMaxConnPerRoute Max number of connections per route to keep open.
Default: 100		 Positive integer.
Ex: 50
httpUserAgent Override User-Agent in request headers. Can help bypass excessively restrictive filters on the HTTP proxy. Default user agent: Wavefront-Proxy/<version>. A string.
Ex: 'Mozilla/5.0'
4.1
idFile Location of the PID file for the wavefront-proxy process.
Default: <dshell>/.id		 Valid path on the local file system. This option is ignored when ephemeral=true.  
jsonListenerPorts TCP ports to listen on for incoming JSON-formatted metrics.
Default: none		 Comma-separated list of available port numbers. Can be a single port.  
listenerIdleConnectionTimeout Close idle inbound connections after specified time in seconds.
Default: 300 		Number of seconds. 4.31
logsIngestionConfigFile The file containing instructions for parsing log data into metrics. See Log Data Metrics Integration.
Default:<wf_config_path>/logsIngestion.yaml 		Valid path on the local file system. 4.1
opentsdbPorts TCP ports to listen on for incoming OpenTSDB-formatted data.
Default: none		 Comma-separated list of available port numbers. Can be a single port.
Ex: 4242
3.1
picklePorts TCP ports to listen on for incoming data in Graphite pickle format (from carbon-relay).
Default: none		 Comma-separated list of available port numbers. Can be a single port.
Ex: 5878
3.20
prefix String to prepend before every metric name. For example, if you set prefix to production, a metric that is sent to the proxy as cpu.loadavg.1m is sent from the proxy to the Operations for Applications service as production.cpu.loadavg.1m.
Default: none		 A lowercase alphanumeric string, with periods separating segments. You do not need to include a trailing period.
Ex: production
Ex: production.nyc.dc1
  
preprocessorConfigFile Path to the optional preprocessor config file containing preprocessor rules for filtering and rewriting metrics.
Default: none		 Valid path on the local file system.
Ex: <wf_config_path>/rules.yaml
 4.1
privateKeyPath Path to PKCS#8 private key file in PEM format. Incoming TLS/SSL connections access this private key. 8.0
privateCertPath Path to X.509 certificate chain file in PEM format. Incoming TLS/SSL connections access this certificate. 8.0
proxyHost HTTP proxy host to be used in configurations when direct HTTP connections to Operations for Applications instances are not possible. Must be used with proxyPort. A string.
Ex: proxy.local
 3.23
proxyname A name unique across your account representing the name of the proxy. The value is used for the source tag of the proxy metrics, such as JVM statistics, per-proxy point rates, and so on. A string containing alphanumeric characters and periods. 12.0
proxyPassword When used with proxyUser, sets credentials to use with the HTTP proxy if the proxy requires authentication. A string.
Ex: validPassword123
3.23
proxyPort HTTP proxy port to be used in configurations when direct HTTP connections to Operations for Applications instances are not possible. Must be used with proxyHost. A port number.
Ex: 8080
3.23
proxyUser When used with proxyPassword, sets credentials to use with the HTTP proxy if the proxy requires authentication. A string.
Ex: validUser
3.23
pushBlockedSamples Number of blocked points to print to the log immediately following each summary line (every 10 flushes). If 0, prints none. If you see a non-zero number of blocked points in the summary lines and want to debug what that data is, set this property to 5.
Default: 5 		0 or a positive integer.
Ex: 5
 
pushFlushInterval Milliseconds to wait between each flush.
Default: 1000		 An integer equal to or greater than 1000.
Ex: 1000
 
pushFlushMaxPoints Maximum number of points to send to the Operations for Applications service during each flush.
Default: 40000		 Positive integer.
Ex: 40000
 
pushFlushMaxHistograms Maximum number of histograms to send to the Operations for Applications service during each flush.
Default: 10000		 Positive integer.
Ex: 10000
6.0
pushFlushMaxSpans Maximum number of spans to send to the Operations for Applications service during each flush.
Default: 5000		 Positive integer.
Ex: 5000
6.0
pushFlushMaxSpanLogs Maximum number of span logs to send to the Operations for Applications service during each flush.
Default: 1000		 Positive integer.
Ex: 1000
6.0
pushListenerHttpBufferSize Maximum allowed request size (in bytes) for incoming HTTP requests on Operations for Applications, OpenTSDB, or Graphite ports.
Default: 16777216 (16MB). 		Ex: 8388608 4.31
pushListenerMaxReceivedLength Maximum line length for received points in plaintext format on Operations for Applications, OpenTSDB, or Graphite ports.
Default: 32KB		 Positive integer.
Ex: 4096
4.31
pushListenerPorts Port to listen on for incoming data. A single port definition can accept both HTTP and TCP data. For HTTP data, make a POST to this proxy port with an empty header, and the line terminated data format. If you want to use HTTPS/TLS, set the tlsPorts, privateKeyPath, and privateCertPath properties as well.
Default: 2878		 Comma-separated list of available port numbers. Can be a single port.
Ex: 2878
Ex: 2878,2879,2880
  
pushLogLevel Frequency to print status information on the data flow to the log. SUMMARY prints a line every 60 flushes, while DETAILED prints a line on each flush. None, SUMMARY, or DETAILED
Ex: SUMMARY
Before 6.0.
pushMemoryBufferLimit Maximum number of points that can stay in memory buffers before spooling to disk. Setting this value lower than default reduces memory usage but forces the proxy to queue points by spooling to disk more frequently, if you have points arriving at the proxy in short bursts. Default: 16 * <pushFlushMaxPoints>
Minimum: <pushFlushMaxPoints>		 Positive integer.
Ex: 640000
 4.1
pushRateLimit Maximum number of points per second to send.
Default: unlimited		 Positive integer.
Ex: 20000
 4.1
pushRateLimitHistograms Maximum number of histograms per second to send.
Default: unlimited		 Positive integer.
Ex: 20000
 6.0
pushRateLimitSpans Maximum number of spans per second to send.
Default: unlimited		 Positive integer.
Ex: 10000
 6.0
pushRateLimitSpanLogs Maximum number of span logs per second to send.
Default: unlimited		 Positive integer.
Ex: 10000
 6.0
pushRateLimitMaxBurstSeconds Max number of burst seconds to allow when rate limiting to smooth out uneven traffic. Set to 1 when doing data backfills.
Default: 10		 Positive integer.
Ex: 20
pushRelayListenerPorts Ports to receive the data sent to the relay. In environments where direct outbound connections to the Operations for Applications instance are not possible, you can use another Wavefront proxy that has outbound access to act as a relay and forward all the data received on that endpoint (from direct data ingestion clients and/or other proxies) to your Operations for Applications instance.
Default: none		 Comma-separated list of available port numbers. Can be a single port.
Ex: 2978
Ex: 2978,2979		 6.0
pushRelayHistogramAggregator If set to true, aggregates the histogram distributions received on the relay port.
Default: false		 Boolean.
Ex: true
 6.0
pushValidationLevel Level of validation to perform on incoming data before sending the data to the Operations for Applications service. If NO_VALIDATION, all data is sent forward. If NUMERIC_ONLY, data is checked to make sure that it is numerical and dropped locally if it is not. NUMERIC_ONLY or NO_VALIDATION
Ex: NUMERIC_ONLY
 
rawLogsMaxReceivedLength Maximum line length for received raw logs.
Default: 4096 		Positive integer.
Ex: 4096
4.31
rawLogsPort TCP port to listen on for log data.
Default: none		 A port number.
Ex: 5045
4.4
retryBackoffBaseSeconds For exponential back-off when retry threads are throttled, the base (a in a^b) in seconds.
Default: 2.0		 Positive number, integer or decimal.
Ex: 2.0
 
retryThreads Number of threads retrying failed transmissions. If no value is specified, defaults to the number of processor cores available to the host or 4, whichever is greater. Every retry thread uses a separate buffer file (capped at 2GB) to persist queued data points, so the number of threads controls the maximum amount of space that the proxy can use to buffer points locally. Positive integer.
Ex: 4
Before 6.0.
server The API URL of the Operations for Applications instance in the format https://<your_instance>.wavefront.com/api/.    
soLingerTime Enables SO_LINGER with the specified linger time in seconds. Set this value to 0 when running in a high-availability configuration under a load balancer.
Default: -1 (disabled).
0 or a positive integer.
Ex: 0 		4.1
splitPushWhenRateLimited Whether to split the push batch size when the push is rejected due to rate limit.
Default: false		 Boolean.
Ex: false
 
tlsPorts Ports to be used for incoming TLS/SSL connections. To set up a port to use TLS/SSL, you specify pushListenerPorts, tlsPorts, privateKeyPath, and privateCertPath. Comma-separated list of ports. Use * to secure all ports. Can be a single port number.
Ex: 4443 		8.0
allow
(Renamed from whiteListRegex in proxy 9.x.)		 Regex pattern (java.util.regex). Input lines are checked against the pattern as they come in and before the prefix is prepended. Only input lines that match are accepted.

Use preprocessor rules for finer-grained control.

 Valid regex pattern.
Ex: ^(production|stage)., which allows points that begin with production. and stage.
3.1
writeHttpJsonListenerPorts Ports to listen on for incoming data from the collected write_http plugin.
Default: none		 Comma-separated list of available port numbers. Can be a single port.
Ex: 4878
3.14
rawLogsHttpBufferSize The maximum request size (in bytes) for incoming HTTP requests with tracing data.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216 		4.38
trafficShaping Enables intelligent traffic shaping based on the data receive rate over the last 5 minutes.
Default: false		 Boolean.
Ex: true		 9.0
trafficShapingQuantile Sets the quantile for traffic shaping.
Default: 75		 An integer.
Ex: 99, so that the 99th percentile of the received rate in the last 5 minutes, will be used as a basis for the rate limiter. 		9.0
trafficShapingHeadroom Sets the headroom multiplier for traffic shaping when there's backlog.
Default: 1.15 (15% headroom) 		Number from 1.0 to 1.99
Ex: 1.05 (5% headroom) 		9.0
sqsBuffer Use AWS SQS for buffering transmissions.
Default: false		 Boolean.
Ex: true 		9.0
sqsQueueNameTemplate The replacement pattern for naming the SQS queues. Ex: wf-proxy-{{id}}-{{entity}}-{{port}}, which results in a queue named wf-proxy-id-points-2878 9.0
sqsQueueIdentifier An identifier for identifying the proxies in SQS. A string
Ex: wavefront 		9.0
sqsQueueRegion The AWS Region name the queue lives in. A string
Ex: us-west-2 		9.0
 click for top of page

Authentication Proxy Properties

Because the proxy is running in your local network by default, communication to the proxy is un-authenticated. If you want to authenticate inbound traffic to the proxy, use the settings in this section.

The Wavefront proxy must authenticate outbound traffic to the Operations for Applications service. See Authenticate Incoming HTTP Requests at the Proxy for step-by-step instructions.

Property Purpose Format and Example
authMethod Authentication is done by a token. Specify the tokens either in the query string ("token=" and "api_key=" parameters) or in the request headers ("X-AUTH-TOKEN: ", "Authorization: Bearer", "Authorization: " headers). TCP streams are disabled when authentication is turned on.
Default: NONE		 Allowed authMethod: NONE, STATIC_TOKEN, HTTP_GET, OAUTH2.
Ex: OAUTH2
authTokenIntrospectionServiceUrl URL for the token introspection endpoint used to validate tokens for incoming HTTP requests. Required when authMethod is OAUTH2 or HTTP_GET.
Default: none
Ex: https://auth.acme.de/api/token/<token>/validate		 Valid URL.
authTokenIntrospectionAuthorizationHeader Optional credentials for use with the token introspection endpoint if the endpoint requires authentication.
Ex: Authorization: Bearer <token>
authResponseRefreshInterval Cache TTL (in seconds) for token validation results (re-authenticate when expired).
Default: 600 		Integer.
Ex: 1200
authResponseMaxTtl Maximum allowed cache TTL (in seconds) for token validation results when the token introspection service is unavailable.
Default: 86400 (1 day)		 Integer
Ex: 3600
authStaticToken Static token that is considered valid for all incoming HTTP requests. Required when authMethod is STATIC_TOKEN.
Default: none 		Valid token
Ex: token1234abcd

OpenTelemetry Proxy Properties

Property Purpose Format /Example
otlpGrpcListenerPorts Ports to receive OpenTelemetry Protobuf data over gRPC.
Default: none
Version: Since 11.0		 Comma-separated list of available port numbers. Can be a single port.
Recommended: 4317
otlpHttpListenerPorts Ports to receive OpenTelemetry Protobuf data over HTTP.
Default: none
Version: Since 11.0		 Comma-separated list of available port numbers. Can be a single port.
Recommended: 4318
otlpResourceAttrsOnMetricsIncluded Set the value to true to include the OpenTelemetry resource attributes that your application sends for metrics data.
Default: false
Version: Since 11.3		 true or false
otlpAppTagsOnMetricsIncluded Set the value to false to exclude the OpenTelemetry resource attributes (application, service.name, shard, and cluster) that your application sends for metrics data.
Default: true
Version: Since 12.0		 true or false
 click for top of page

Tracing Proxy Properties

Property Purpose Format /Example
traceAlwaysSampleErrors Always sample spans with an error tag ignoring other sampling configuration.
Default: true		 Boolean.
Ex: false
traceJaegerHttpListenerPorts TCP ports to receive Jaeger Thrift formatted data via HTTP. The data is then sent to the Operations for Applications service in Operations for Applications span format.
Default: none
Version: Since 6.0		 Comma-separated list of available port numbers. Can be a single port.
traceJaegerListenerPorts TCP ports to receive Jaeger Thrift formatted data via TChannel. The data is then sent to the Operations for Applications service in Operations for Applications span format.
Default: none 		Comma-separated list of available port numbers. Can be a single port.
traceJaegerGrpcListenerPorts Ports to receive Jaeger Protobuf formatted data over gRPC.
Default: none
Version: Since 9.0		 Comma-separated list of available port numbers. Can be a single port.
customTracingListenerPorts TCP ports to receive spans and derive RED metrics from the SDKs that send raw data to the Operations for Applications service.
Default: None.
Version: Since 6.0 		Comma-separated list of available port numbers. Can be a single port.
traceListenerMaxReceivedLength Maximum line length for received spans and span logs.
Default: 1MB		 Integer
Ex: 1048576
traceListenerPorts TCP ports that listen to incoming spans from the Operations for Applications SDKs that collect metrics and histograms, and SDKs for sending raw data.
Default: none		 Comma-separated list of available port numbers. Can be a single port.
Ex: 2878
Ex: 2878,2879
traceSamplingDuration Minimum duration of the tracing spans that can be sent to the Operations for Applications service for trace data sampling.
Default: 0 (send all generated spans). 		Number of milliseconds.
Ex: 45
traceSamplingRate Percentage of all generated spans to send to the Operations for Applications service for trace data sampling.
Default: 1.0 (send all generated spans). 		Number from 0.0 to 1.0.
Ex: .1
traceZipkinListenerPorts TCP ports to listen on for Zipkin formatted data.
Recommended: The default Zipkin Collector port (9411).
Default: none		 Comma-separated list of available port numbers. Can be a single port.
customTracingApplicationName Custom application name for spans received on the customTracingListenerPorts that don't have the application tag.
Default: defaultApp
Version: Since 9.0		 Ex: MyApplication
customTracingServiceName Custom service name for spans received on the customTracingListenerPorts that don't have the service tag.
Default: defaultService
Version: Since 9.0		 Ex: MyService
traceJaegerApplicationName Custom application name for traces received on Jaeger's traceJaegerListenerPorts or traceJaegerHttpListenerPorts. Ex: MyJaegerDemo
traceZipkinApplicationName Custom application name for traces received on Zipkin's traceZipkinListenerPorts. Ex: MyZipkinDemo
traceDerivedCustomTagKeys Comma separated list of custom tag keys to include as metric tags for the derived RED (Request, Error, Duration) metrics. Applicable to Jaeger and Zipkin integration only. Ex: tenant,env,location
traceListenerHttpBufferSize The maximum request size (in bytes) for incoming HTTP requests with tracing data.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216
 click for top of page

Histogram Configuration Properties

Operations for Applications supports additional histogram configuration properties, shown in the following table. The requirements on the state directory and the effect of the two persist properties listed at the bottom of the table.

PropertyDescriptionFormat
histogramAccumulatorResolveInterval Interval in milliseconds to write back accumulation changes from memory cache to disk. Only applicable when memory cache is enabled. Increasing this setting reduces storage IO pressure but might increase heap memory use.
Default: 100		 Positive integer.
histogramAccumulatorFlushInterval Interval in milliseconds to check for histograms to be sent to a Operations for Applications service according to their histogramMinuteFlushSecs settings.
Default: 1000		 Positive integer.
histogramAccumulatorFlushMaxBatchSize Max number of histograms to move to the outbound queue in one flush.
Default: no limit		 Positive integer.
histogramDayAccumulators Number of accumulators per day port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores.
Default: 2		 Positive integer.
histogramDayAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins.
Default: 100000		 Positive integer.
histogramDayAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramDayAvgDigestBytes Average number of bytes in an encoded distribution/accumulation.
Default: 32 + histogramDayCompression * 7		 Positive integer.
histogramDayAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags.
Default: 150		 Positive integer.
histogramDayCompression A bound on the number of centroids per histogram.
Default: 100		 Positive integer in the interval [20;1000].
histogramDayFlushSecs Time-to-live, in seconds, for a day granularity accumulation on the proxy (before the intermediary is sent to the Operations for Applications service).
Default: 18000 (5 hours). 		Positive integer.
histogramDayListenerPorts TCP ports to listen on for histograms to be aggregated by day.
Default: 40003		 Comma-separated list of ports.
histogramDayMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series).
Default: false		 Boolean.
histogramDistAccumulators Number of accumulators per distribution port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores.
Default: number of available CPU cores 		Positive integer.
histogramDistAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramDistAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins.
Default: 100000		 Positive integer.
histogramDistAvgDigestBytes Average number of bytes in an encoded distribution/accumulation.
Default: 32 + histogramDistCompression * 7		 Positive integer.
histogramDistCompression A bound on the number of centroids per histogram.
Default: 100		 Positive integer in the interval [20;1000].
histogramDistFlushSecs Number of seconds to keep a new distribution bin open for new samples, before the intermediary is sent to the Operations for Applications service.
Default: 70		 Positive integer.
histogramDistListenerPorts TCP ports to listen on for ingesting histogram distributions.
Default: 40000		 Comma-separated list of ports. Can be a single port.
histogramDistMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (Aggregating more than 1 distribution per second per time series).
Default: false		 Boolean.
histogramDistAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags.
Default: 150		 Positive integer.
histogramHttpBufferSize Since 4.40. The maximum request size (in bytes) for incoming HTTP requests on histogram ports.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216
histogramHourAccumulators Number of accumulators per hour port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores.
Default: 2		 Positive integer.
histogramHourAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins.
Default: 100000		 Positive integer.
histogramHourAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramHourAvgDigestBytes Average number of bytes in an encoded distribution/accumulation.
Default: 32 + histogramMinuteCompression * 7		 Positive integer.
histogramHourAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags.
Default: 150		 Positive integer.
histogramHourCompression A bound on the number of centroids per histogram.
Default: 100		 Positive integer in the interval [20;1000].
histogramHourFlushSecs Time-to-live, in seconds, for an hour granularity accumulation on the proxy (before the intermediary is sent to the Operations for Applications service).
Default: 4200		 Positive integer.
histogramHourListenerPorts TCP ports to listen on for histograms to be aggregated by hour.
Default: 40002		 Comma-separated list of ports. Can be a single port.
histogramHourMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series).
Default: false		 Boolean.
histogramMaxReceivedLength Maximum line length for received histogram points.
Default: 65536		 Positive integer.
histogramMinuteAccumulatorPersisted Whether to persist accumulation state. If true, all histograms are written to disk immediately if memory cache is disabled, or every histogramAccumulatorResolveInterval seconds if memory cache is enabled. If false, up to histogramMinuteFlushSecs seconds worth of histograms may be lost on proxy shutdown. Boolean.
Ex: false
histogramMinuteListenerPorts TCP ports to listen on for histograms to be aggregated by minute.
Default: 40001		 Comma-separated list of ports. Can be a single port.
histogramMinuteAccumulators Number of accumulators per minute port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores.
Default: 2		 Positive integer.
histogramMinuteAccumulatorSize Expected upper bound of concurrent accumulations. Should be approximately the number of time series * 2 (use a higher multiplier if out-of-order points more than 1 minute apart are expected). Setting this value too high results in excessive disk space usage, setting this value too low may cause severe performance issues.
Default: 100000		 Positive integer.
histogramMinuteAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags.
Default: 150		 Positive integer.
histogramMinuteCompression A bound on the number of centroids per histogram.
Default: 32		 Positive integer in the interval [20;1000].
histogramMinuteFlushSecs Time-to-live, in seconds, for a minute granularity accumulation on the proxy (before the intermediary is sent to the Operations for Applications service).
Default: 70		 Positive integer.
histogramMinuteMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series).
Default: false		 Boolean.
histogramMinuteAvgDigestBytes Average number of bytes in an encoded distribution/accumulation.
Default: 32 + histogramMinuteCompression * 7		 Positive integer.
histogramProcessingQueueScanInterval Interval in milliseconds between checks for new entries in the processing queue.
Default: 20		 Positive integer.
histogramReceiveBufferFlushInterval Sets maximum time in milliseconds that incoming points can stay in the receive buffer when incoming traffic volume is very low.
Default: 100		 Positive integer.
histogramStateDirectory Directory for persistent proxy state, must be writable. Before being flushed to the Operations for Applications service, histogram data is persisted on the filesystem where the Wavefront proxy runs. If the files are corrupted or the files in the directory can't be accessed, the proxy reports the problem in its log and fails back to using in-memory structures. In this mode, samples can be lost if the proxy terminates without draining its queues.
Default: /var/spool/wavefront-proxy. 		A valid path on the local file system.
persistAccumulator Whether to persist accumulation state. We suggest keeping this setting enabled unless you are not using hour and day level aggregation and consider losing up to 1 minute worth of data during proxy restarts acceptable.
Default: true 		Boolean.
persistMessages Whether to persist received metrics to disk.
Default: true 		Boolean.
pushRelayHistogramAggregatorAccumulatorSize Since 6.0. Max number of concurrent histogram to accumulations at the relay ports. The value is approximately the number of time series * 2 (use a higher multiplier if out-of-order points more than 1 bin apart are expected). Setting this value too high will cause excessive disk space usage, setting this value too low may cause severe performance issues. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 32		 Positive integer.
pushRelayHistogramAggregatorCompression Since 6.0. Number of centroids per histogram. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 32		 Must be between 20 and 1000.
Ex: 40
pushRelayHistogramAggregatorFlushSecs Since 6.0. Interval in milliseconds to check for histograms that have accumulated at the relay ports before sending data to the Operations for Applications service. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 70		 Number of milliseconds.
Ex: 80
 click for top of page

Authenticate Incoming HTTP Requests at the Proxy

The Wavefront proxy can be configured to authenticate inbound traffic. Below are the steps for setting up the different supported authentication methods.

Authentication Method Static Token

Required properties: authStaticToken

In order for data sent via HTTP to the proxy to be accepted, the request will need to include a token matching whatever is configured for authStaticToken. The token can be specified in the header or in the query string.

StepExample
1. Set authMethod to STATIC_TOKEN. authMethod=STATIC_TOKEN
2. Set authStaticToken to a value that will be used as the authentication token. authStaticToken=token1234abcd
3. Ensure that valid data sent to the proxy has the appropriate token included with the request.  

Authentication Method HTTP_Get or OAUTH2

Required properties: authTokenIntrospectionServiceUrl

Optional properties: authTokenIntrospectionAuthorizationHeader, authResponseRefreshInterval, authResponseMaxTtl

If you have a service that can validate tokens, use this approach.

  • Use OAUTH2 if your Introspection Service is RFC7662 compliant.
  • Otherwise, use HTTP_GET.

For either method, the service must return a 2xx code for valid tokens. According to the standard, OAUTH2-compliant services return a JSON object that contains the active field.

StepExample
1. Set authMethod to HTTP_GET or OAUTH2. authMethod=OAUTH2
2. Set authTokenIntrospectionServiceUrl to the appropriate token validation endpoint for your introspection service. Use {{token}} as the placeholder for the token. authTokenIntrospectionServiceUrl=
https://auth.acme.corp/api/token/{{token/validate}}
3. If the token validation endpoint requires authentication, specify authTokenIntrospectionAuthorizationHeader. authTokenIntrospectionAuthorizationHeader=
Authorization: Bearer token123xyz
4. Optionally, set authResponseRefreshInterval to specify how long to cache token validation results, in seconds, before re-authenticating against the introspection service. Default is 600 seconds (10 minutes). authResponseRefreshInterval=300
5. Optionally, set authResponseMaxTtl to specify the maximum amount of time, in seconds, to cache token validation results if the introspection service cannot be reached. Default is 86400 seconds (1 day). authResponseMaxTtl=21600
6. Ensure that valid data sent to the proxy has the appropriate token included with the request.  
 click for top of page

Troubleshooting

proxies