Proxy configuration properties and advanced install info

You can configure proxies using a configuration file, and you can perform advanced installation management such as installing proxies in a container.

In addition to the proxy configuration properties discussed here you can also use proxy preprocessor rules. These rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.

Proxy Configuration Properties

The main Wavefront proxy configuration file is maintained in <wavefront_config_path>/wavefront.conf (<wf_config_path>/wavefront.conf). The configuration file offers many options for changing how the proxy processes your data. There are optional configuration files for rewriting metrics and parsing log data. The default values work well in many cases, but you can adjust them as needed. After changing a configuration option, restart the proxy service.

Paths

In this section, file paths use the following conventions and values:

  • <wavefront_config_path>
    • Linux - /etc/wavefront/wavefront-proxy
    • Mac - /usr/local/etc/wavefront/wavefront-proxy
    • Windows - C:\Program Files (x86)\Wavefront\conf
  • <wavefront_log_path>
    • Linux - /var/log/wavefront
    • Mac - /usr/local/var/log/wavefront
    • Windows - C:\Program Files (x86)\Wavefront
  • <wavefront_spool_path>
    • Linux - /var/spool/wavefront-proxy
    • Mac - /usr/local/var/spool/wavefront-proxy
    • Windows - C:\Program Files (x86)\Wavefront\bin

General Proxy Properties and Examples

This section lists:

  • General proxy configuration properties
  • Metrics proxy configuration properties
  • Tracing proxy configuration properties

See the Histogram Configuration Properties table below for properties specific to histogram distributions.

Property Purpose Format and Example Since
agentMetricsPointTags Point tags and their values to be passed along with ~agent./ metrics.
Default: None.		 Comma-separated list of key-value pairs.
Ex: dc=west,env=prod		 3.24
blacklistRegex Regex pattern (java.util.regex) that input lines must match to be filtered out. Input lines are checked against the pattern as they come in and before the prefix is prepended. Valid regex pattern.
Ex: Filter out points that begin with qa., development., or test.: ^(qa|development|test).		 3.1
blockedPointsLoggerName Controls how the blocked points are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints. 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
blockedHistogramsLoggerName Controls how the blocked histograms are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints. 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
blockedSpansLoggerName Controls how the blocked spans are logged. For details, see Blocked Data Log.
Default: RawBlockedPoints. 		Logger name for blocked points.
Ex: RawBlockedPoints		 6.0
buffer Location of buffer files for saving failed transmissions for retry. Valid path on the local file system.
Ex: <wf_spool_path>/buffer		 The start
customSourceTags Point tag keys to use as 'source' if no 'source' or 'host' field is present.
Default: fqdn, hostname.		 Comma-separated list of point tag keys.
Ex: fqdn, hostname		 3.14
dataBackfillCutoffHours The cut-off point for what is considered a valid timestamp for back-dated points. We do not recommend setting this value larger than 1 year unless backfilling or migrating historic data.
Default: 8760 (1 year), so all points older than 1 year are rejected.		 Positive integer.
Ex: 8760		 4.1
deltaCountersAggregationListenerPorts Port to listen to the Wavefront-formatted delta counter data. Other data formats are rejected at this port. Pre-aggregating delta counters at the proxy, helps reduce the outbound point rate. Use this property in conjunction with deltaCountersAggregationIntervalSeconds to limit the number of points per second for delta counters.
Default: none.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 12878
Ex: 12878,12879		 6.0
deltaCountersAggregationIntervalSeconds Interval between flushing aggregating delta counters to Wavefront. Use this property in conjunction with deltaCountersAggregationListenerPorts to send points to the port(s) in batches, thereby limiting the number of points per second.
Default: 30 seconds. 		Number of seconds.
Ex: 45		 6.0
ephemeral Whether to automatically clean up old and orphaned proxy instances from the Wavefront Proxies page. We recommend enabling ephemeral mode if you're running the proxy in a container that may be frequently spun down and recreated.
Default: true. 		Boolean
Ex: false 		3.14
fileBeatPort TCP port to listen on for Filebeat data.
Default: 5044.		 A port number.
Ex: 5044 		4.1
flushThreads Number of threads that flush data to the server. Setting this value too high results in sending batches that are too small to the Wavefront server and wasting connections. Values between 6 and 16 are a good starting point. This setting is per listening port.
Default: The number of available processors (min 4).		 Positive integer.
Ex: 16		 3.14
graphiteDelimiters Characters that should be replaced by dots, in case they were escaped within Graphite and collectd before sending. A common delimiter is the underscore character; so if you extract a hostname field with the value web04_www, it is changed to web04.www. A concatenation of delimiter characters, without any separators.  
graphiteFormat Indexes of fields within Graphite and collectd metric names that correspond to a hostname. For example, if your metrics have the format: collectd.prod.www04.cpu.loadavg.1m, specify the 3rd and 2nd indexes (www04.prod) to be extracted and treated as the hostname. The remainder collectd.cpu.loadavg.1m is treated as the metric name. Comma-separated list of indexes.
Ex: 4, 2, 5 Ex: 3 		 
graphitePorts TCP ports to listen on for Graphite data. Define which of the segments in your Graphite metrics map to a hostname in the graphiteFormat property. Default: 2003. Comma-separated list of available port numbers. Can be a single port.
Ex: 2003
Ex: 2003, 2004 		 
gzipCompression If set to true, metric traffic from the proxy to the Wavefront endpoint is gzip-compressed.
Default: true.		 true or false
Ex: true		  
gzipCompressionLevel Sets the gzip compression level if gzipCompression is enabled. The level vary from, 1 to 9. Higher compression levels slightly reduces the volume of traffic between the proxy and Wavefront, but uses more CPU.
Default: 4.		 Positive integer ranging from 1 to 9.
Ex: 4		 6.0
Histogram configuration properties Properties specific to histogram distributions, listed in a separate table below. 4.31
hostname A name unique across your account representing the machine that the proxy is running on. The hostname is not used to tag your metrics; rather, it's used to tag proxy metrics, such as JVM statistics, per-proxy point rates, and so on. A string containing alphanumeric characters and periods.  
httpConnectTimeout HTTP connect timeout (in milliseconds).
Default: 5000 (5s).		 Positive integer.
Ex: 5000
4.1
httpRequestTimeout HTTP request timeout (in milliseconds). We do not recommend setting this value to be higher than 20000. Recommended value for most configurations is 10000 (10 seconds).
Default: 10000 (10s).		 Positive integer.
Ex: 10000
4.1
httpUserAgent Override User-Agent in request headers. Can help bypass excessively restrictive filters on the HTTP proxy. Default user agent: Wavefront-Proxy/<version>. A string.
Ex: 'Mozilla/5.0'
4.1
idFile Location of the PID file for the wavefront-proxy process.
Default: <dshell>/.id		 Valid path on the local file system. This option is ignored when ephemeral=true.  
jsonListenerPorts TCP ports to listen on for incoming JSON-formatted metrics.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.  
listenerIdleConnectionTimeout Close idle inbound connections after specified time in seconds.
Default: 300 		Number of seconds. 4.31
logsIngestionConfigFile The file containing instructions for parsing log data into metrics. See Log Data Metrics Integration. Default: <cfg_path>/logsIngestion.yaml. Valid path on the local file system. 4.1
opentsdbPorts TCP ports to listen on for incoming OpenTSDB-formatted data.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 4242
3.1
picklePorts TCP ports to listen on for incoming data in Graphite pickle format (from carbon-relay).
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 5878
3.20
prefix String to prepend before every metric name. For example, if you set prefix to 'production', a metric that is sent to the proxy as cpu.loadavg.1m is sent from the proxy to Wavefront as production.cpu.loadavg.1m. You can include longer prefixes such as production.nyc.dc1.
Default: None.		 A lowercase alphanumeric string, with periods separating segments. You do not need to include a trailing period.
Ex: production
Ex: production.nyc.dc1
  
preprocessorConfigFile Path to the optional preprocessor config file containing preprocessor rules for filtering and rewriting metrics.
Default: None.		 Valid path on the local file system.
Ex: <cfg_path>/rules.yaml
 4.1
proxyHost HTTP proxy host to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyPort. A string.
Ex: proxy.local
 3.23
proxyPassword When used with proxyUser, sets credentials to use with the HTTP proxy if the proxy requires authentication. A string.
Ex: validPassword123
3.23
proxyPort HTTP proxy port to be used in configurations when direct HTTP connections to Wavefront servers are not possible. Must be used with proxyHost. A port number.
Ex: 8080
3.23
proxyUser When used with proxyPassword, sets credentials to use with the HTTP proxy if the proxy requires authentication. A string.
Ex: validUser
3.23
pushBlockedSamples Number of blocked points to print to the log immediately following each summary line (every 10 flushes). If 0, print none. If you see a non-zero number of blocked points in the summary lines and want to debug what that data is, set this property to 5.
Default: 5 		0 or a positive integer.
Ex: 5
 
pushFlushInterval Milliseconds to wait between each flush to Wavefront.
Default: 1000.		 An integer equal to or greater than 1000.
Ex: 1000
 
pushFlushMaxPoints Maximum number of points to send to Wavefront during each flush.
Default: 40,000.		 Positive integer.
Ex: 40000
 
pushFlushMaxHistograms Maximum number of histograms to send to Wavefront during each flush.
Default: 10,000.		 Positive integer.
Ex: 10000
6.0
pushFlushMaxSpans Maximum number of spans to send to Wavefront during each flush.
Default: 5,000.		 Positive integer.
Ex: 5000
6.0
pushFlushMaxSpanLogs Maximum number of span logs to send to Wavefront during each flush.
Default: 1,000.		 Positive integer.
Ex: 1000
6.0
pushListenerHttpBufferSize Maximum allowed request size (in bytes) for incoming HTTP requests on Wavefront, OpenTSDB, or Graphite ports.
Default: 16777216 (16MB). 		Ex: 8388608 4.31
pushListenerMaxReceivedLength Maximum line length for received points in plaintext format on Wavefront, OpenTSDB, or Graphite ports.
Default: 4096		 Positive integer.
Ex: 4096
4.31
pushListenerPorts Port to listen on for incoming data. A single port definition can accept both HTTP and TCP data. For HTTP data, make a POST to this proxy port with an empty header, and the line terminated data format.
Default: 2878.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 2878
Ex: 2878,2879,2880
  
pushLogLevel Frequency to print status information on the data flow to the log. SUMMARY prints a line every 60 flushes, while DETAILED prints a line on each flush. None, SUMMARY, or DETAILED
Ex: SUMMARY
Before 6.0.
pushMemoryBufferLimit Maximum number of points that can stay in memory buffers before spooling to disk. Setting this value lower than default reduces memory usage but forces the proxy to queue points by spooling to disk more frequently, if you have points arriving at the proxy in short bursts. Default: 16 * pushFlushMaxPoints. Minimum: pushFlushMaxPoints. Positive integer.
Ex: 640000
 4.1
pushRateLimit Maximum number of points per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 20000
 4.1
pushRateLimitHistograms Maximum number of histograms per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 20000
 6.0
pushRateLimitSpans Maximum number of spans per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 10000
 6.0
pushRateLimitSpanLogs Maximum number of span logs per second to send to Wavefront.
Default: unlimited.		 Positive integer.
Ex: 10000
 6.0
pushRelayListenerPorts Ports to receive the data sent to the relay. In environments where direct outbound connections to Wavefront servers are not possible, you can use another Wavefront proxy that has outbound access to act as a relay and forward all the data received on that endpoint (from direct data ingestion clients and/or other proxies) to Wavefront servers.
Default: none.		 Comma-separated list of available port numbers. Can be a single port. Ex: 2978
Ex: 2978,2979		 6.0
pushRelayHistogramAggregator If set to true, aggregates the histogram distributions received on the relay port.
Default: false.		 true or false
Ex: true
 6.0
pushValidationLevel Level of validation to perform on incoming data before sending the data to Wavefront. If NO_VALIDATION, all data is sent forward. If NUMERIC_ONLY, data is checked to make sure that it is numerical and dropped locally if it is not. NUMERIC_ONLY or NO_VALIDATION
Ex: NUMERIC_ONLY
 
rawLogsMaxReceivedLength Maximum line length for received raw logs.
Default: 4096. 		Positive integer.
Ex: 4096
4.31
rawLogsPort TCP port to listen on for log data.
Default: 5045.		 A port number.
Ex: 5045
4.4
retryBackoffBaseSeconds For exponential back-off when retry threads are throttled, the base (a in a^b) in seconds.
Default: 2.0.		 Positive number, integer or decimal.
Ex: 2.0
 
retryThreads Number of threads retrying failed transmissions. If no value is specified, defaults to the number of processor cores available to the host or 4, whichever is greater. Every retry thread uses a separate buffer file (capped at 2GB) to persist queued data points, so the number of threads controls the maximum amount of space that the proxy can use to buffer points locally. Positive integer.
Ex: 4
Before 6.0.
server The API URL of the Wavefront server in the format https://<wf_instance>.wavefront.com/api/.    
soLingerTime Enable SO_LINGER with the specified linger time in seconds. Set this value to 0 when running in a high-availability configuration under a load balancer.
Default: -1 (disabled).
0 or a positive integer.
Ex: 0 		4.1
splitPushWhenRateLimited Whether to split the push batch size when the push is rejected by Wavefront due to rate limit.
Default: false.		 true or false
Ex: false
 
whitelistRegex Regex pattern (java.util.regex). Input lines are checked against the pattern as they come in and before the prefix is prepended. Only input lines that match are accepted. Valid regex pattern.
Ex: ^(production|stage).
Allows points that begin with production. and stage.
3.1
writeHttpJsonListenerPorts Ports to listen on for incoming data from the collected write_http plugin.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 4878
3.14
rawLogsHttpBufferSize The maximum request size (in bytes) for incoming HTTP requests with tracing data.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216 		4.38

Tracing Proxy Properties and Examples

Property Purpose Format /Example Since
traceJaegerHttpListenerPorts TCP ports to receive Jaeger Thrift formatted data via HTTP. The data is then sent to Wavefront in Wavefront span format.
Default: None.		 Comma-separated list of available port numbers. Can be a single port. 6.0
traceJaegerListenerPorts TCP ports to receive Jaeger Thrift formatted data via TChannel. The data is then sent to Wavefront in Wavefront span format.
Default: None. 		Comma-separated list of available port numbers. Can be a single port. 4.31
customTracingListenerPorts TCP ports to receive spans and derive RED metrics from the SDKs that send raw data to Wavefront.
Default: None.		 Comma-separated list of available port numbers. Can be a single port. 6.0
traceListenerPorts TCP ports to listen on for incoming spans.
Default: None.		 Comma-separated list of available port numbers. Can be a single port.
Ex: 30000
Ex: 30000, 30001
 4.31
traceSamplingDuration Minimum duration of the tracing spans that can be sent to Wavefront for trace data sampling.
Default: 0 (send all generated spans). 		Number of milliseconds.
Ex: 45
 4.34
traceSamplingRate Percentage of all generated spans to send to Wavefront for trace data sampling.
Default: 1.0 (send all generated spans). 		Number from 0.0 to 1.0.
Ex: .1
 4.34
traceZipkinListenerPorts TCP ports to listen on for Zipkin formatted data. Recommended: The default Zipkin Collector port (9411).
Default: None.		 Comma-separated list of available port numbers. Can be a single port. 4.35
traceJaegerApplicationName Custom application name for traces received on Jaeger's traceJaegerListenerPorts or traceJaegerHttpListenerPorts. traceJaegerApplicationName=MyJaegerDemo 4.38
traceZipkinApplicationName Custom application name for traces received on Zipkin's traceZipkinListenerPorts. traceZipkinApplicationName=MyZipkinDemo 4.38
traceDerivedCustomTagKeys Comma separated list of custom tag keys to include as metric tags for the derived RED (Request, Error, Duration) metrics. Applicable to Jaeger and Zipkin integration only. traceDerivedCustomTagKeys=tenant, env, location 4.38
traceListenerHttpBufferSize The maximum request size (in bytes) for incoming HTTP requests with tracing data.
Default: 16MB		 Buffer size in bytes.
Ex: 16777216 		4.37

Histogram Configuration Properties

Wavefront supports additional histogram configuration properties, shown in the following table. Note the requirements on the state directory and the effect of the two persist properties listed at the bottom of the table.

PropertyDescriptionFormat
histogramStateDirectory Directory for persistent proxy state, must be writable. Before being flushed to Wavefront, histogram data is persisted on the filesystem where the Wavefront proxy resides. If the files are corrupted or the files in the directory can't be accessed, the proxy reports the problem in its log and fails back to using in-memory structures. In this mode, samples can be lost if the proxy terminates without draining its queues. Default: /var/spool/wavefront-proxy. A valid path on the local file system.
persistAccumulator Whether to persist accumulation state. We suggest keeping this setting enabled unless you are not using hour and day level aggregation and consider losing up to 1 minute worth of data during proxy restarts acceptable. Default: true. .
persistMessages Whether to persist received metrics to disk. Default: true. Boolean.
histogramAccumulatorResolveInterval Interval in milliseconds to write back accumulation changes from memory cache to disk. Only applicable when memory cache is enabled. Increasing this setting reduces storage IO pressure but might increase heap memory use. Default: 100. Positive integer.
histogramAccumulatorFlushInterval Interval in milliseconds to check for histograms that need to be sent to Wavefront according to their histogramMinuteFlushSecs settings. Default: 1000. Positive integer.
histogramAccumulatorFlushMaxBatchSize Max number of histograms to move to the outbound queue in one flush. Default: no limit. Positive integer.
histogramMaxReceivedLength Maximum line length for received histogram points. Default: 65536. Positive integer.
histogramReceiveBufferFlushInterval Sets maximum time in milliseconds that incoming points can stay in the receive buffer when incoming traffic volume is very low. Default: 100. Positive integer.
histogramProcessingQueueScanInterval Interval in milliseconds between checks for new entries in the processing queue. Default: 20. Positive integer.
histogramMinuteListenerPorts TCP ports to listen on for histograms to be aggregated by minute. Default: 40001. Comma-separated list of ports. Can be a single port.
histogramMinuteAccumulators Number of accumulators per minute port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. Positive integer.
histogramMinuteFlushSecs Time-to-live, in seconds, for a minute granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 70. Positive integer.
histogramMinuteAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramMinuteCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval. [20;1000].
histogramMinuteMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. Boolean.
histogramMinuteAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramMinuteCompression * 7 Positive integer.
histogramMinuteAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramHourListenerPorts TCP ports to listen on for histograms to be aggregated by hour. Default: 40002. Comma-separated list of ports. Can be a single port.
histogramHourAccumulators Number of accumulators per hour port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. Positive integer.
histogramHourFlushSecs Time-to-live, in seconds, for an hour granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 4200. Positive integer.
histogramHourAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramHourCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval [20;1000].
histogramHourMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. Boolean.
histogramHourAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramMinuteCompression * 7 Positive integer.
histogramHourAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramDayListenerPorts TCP ports to listen on for histograms to be aggregated by day. Default: 40003. Comma-separated list of ports.
histogramDayAccumulators Number of accumulators per day port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: 2. Positive integer.
histogramDayFlushSecs Time-to-live, in seconds, for a day granularity accumulation on the proxy (before the intermediary is sent to Wavefront). Default: 18000 (5 hours). Positive integer.
histogramDayAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramDayCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval [20;1000].
histogramDayMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (more than 1 point per second per time series). Default: false. Boolean.
histogramDayAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramDayCompression * 7 Positive integer.
histogramDayAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
histogramDistListenerPorts TCP ports to listen on for ingesting histogram distributions. Default: 40000. Comma-separated list of ports. Can be a single port.
histogramDistAccumulators Number of accumulators per distribution port. In high traffic environments we recommend that the total number of accumulators per proxy across all utilized ports does not exceed the number of available CPU cores. Default: number of available CPU cores. Positive integer.
histogramDistFlushSecs Number of seconds to keep a new distribution bin open for new samples, before the intermediary is sent to Wavefront. Default: 70. Positive integer.
histogramDistAccumulatorSize Expected upper bound of concurrent accumulations: ~ #time series * #parallel reporting bins. Default: 100000. Positive integer.
histogramDistCompression A bound on the number of centroids per histogram. Default: 100. Positive integer in the interval [20;1000].
histogramDistMemoryCache Enabling memory cache reduces I/O load with fewer time series and higher frequency data (Aggregating more than 1 distribution per second per time series). Default: false. Boolean.
histogramDistAvgDigestBytes Average number of bytes in an encoded distribution/accumulation. Default: 32 + histogramDistCompression * 7 Positive integer.
histogramDistAvgKeyBytes Average number of bytes in a UTF-8 encoded histogram key. Concatenation of metric, source, and point tags. Default: 150. Positive integer.
pushRelayHistogramAggregatorFlushSecs Since 6.0. Interval in milliseconds to check for histograms that have accumulated at the relay ports before sending data to Wavefront. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 70.		 Number of milliseconds.
Ex: 80
pushRelayHistogramAggregatorCompression Since 6.0. Number of centroids per histogram. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 32.		 Must be between 20 and 1000.
Ex: 40
pushRelayHistogramAggregatorAccumulatorSize Since 6.0. Max number of concurrent histogram to accumulations at the relay ports. The value is approximately the number of time series * 2 (use a higher multiplier if out-of-order points more than 1 bin apart are expected). Setting this value too high will cause excessive disk space usage, setting this value too low may cause severe performance issues. Only applicable if the pushRelayHistogramAggregator is set to true.
Default: 32.		 Positive integer.
histogramHttpBufferSize Since 4.40. The maximum request size (in bytes) for incoming HTTP requests on histogram ports.
Default: 16MB.		 Buffer size in bytes.
Ex: 16777216

Data Buffering

If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:

2013-11-18 18:02:35,061 WARN  [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]

By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:

2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared

Logging

The Wavefront proxy supports two log files: proxy log and blocked point log.

To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. Configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.

If you’re using proxies in containers, you can mount the log files, as discussed below.

Proxy Log

By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.

If you want to set logs for Jaeger and Zipkin integrations, see Logging for Jaeger and Zipkin.

Blocked Data Log

You can log all the raw blocked data separately or log different entities into their separate log files.

  • Log the block data separately
    Follow these steps:
    1. Open the <wavefront_config_path>/log4j2.xml configuration file.
    2. To log all the block data, uncomment the corresponding section. 
       <AsyncLogger name="RawBlockPoints" level="WARN" additivity="false">
    <AppenderRef ref="BlockedPointsFile" />
 </AsyncLogger>

      By default, blocked point entries are logged to the <wavefront_log_path>/wavefront-blocked-points.log file and the log file is rolled over every day when its size reaches 100MB. When there are 31 log files, older files are deleted. You can customize the configurations to suit your environment.

  • Set up separate log files for blocked entities
    Follow these steps:
    1. Uncomment or add the configurations under Appenders and Loggers in the <wavefront_config_path>/log4j2.xml configuration file. 
         <!-- Log the blocked histograms. If you don't need a separate log file for it, 
   don't add this configuration to the file.-->
   <AsyncLogger name="RawBlockedHistograms" level="WARN" additivity="false">
      <AppenderRef ref="[Enter_Your_File_Name]"/>
  </AsyncLogger>
                
  <!-- Logs the blocked points for spans. If you don't need a separate log file for it, 
  don't add this configuration to the file.-->
  <AsyncLogger name="RawBlockedSpans" level="WARN" additivity="false">
      <AppenderRef ref="[Enter_Your_File_Name]"/>
  </AsyncLogger>
         
  <AsyncLogger name="RawBlockPoints" level="WARN" ADDITIVITY="FALSE"/> 
	   <AppenderRef ref=”BlockedPointsFile”/>
  </AsyncLogger>
    2. Add the names of the block points, which you uncommented in the log4j2.xml file, to the <wavefront_config_path>/wavefront.conf file.
      Example: 
         blockedPointsLoggerName = RawBlockedPoints
          
   # Add this if you added the appender for histograms in the log4j2.xml file.
   blockedHistogramsLoggerName = RawBlockedHistograms (RawBlockedPoints by default)
          
   # Add this if you added the appender for spans in the log4j2.xml file.
   blockedSpansLoggerName = RawBlockedSpans (RawBlockedPoints by default)

Configuring a Proxy in a Container

You can use the in-product Docker with cAdvisor or Kubernetes integration if you want to set up a proxy in a container. You can then customize that proxy.

Proxy Versions for Containers

For containers, the proxy image version is determined by the image property in the configuration file. You can set this to image: wavefronthq/proxy:latest, or specify a proxy version explicitly. The proxies are not stateful. Your configuration is managed in your yaml file. It’s safe to use proxy:latest – we ensure that proxies are backward compatible.

Restricting Memory Usage for the Container

To restrict memory usage of the container using Docker, you need to add a JAVA_HEAP_USAGE environment variable and restrict memory using the -m or --memory options for the docker run command. The container memory contraint should be at least 350mb larger than the JAVA_HEAP_USAGE environment variable.

To restrict a container’s memory usage to 2g with Docker run:

docker run -d --name wavefront-proxy ... -e JAVA_HEAP_USAGE="1650m" -m 2g ...

To limit memory usage of the container in Kubernetes use the resources.limits.memory property of a container definition. See the Kubernetes doc.

Customizing Proxy Settings for Docker

When you run a Wavefront proxy inside a Docker container, you can tweak proxy configuration settings that are properties in the wavefront.conf file directly from the Docker run command. You use the WAVEFRONT_PROXY_ARGS environment variable and pass in the property name as a long form argument, preceded by --.

For example, add -e WAVEFRONT_PROXY_ARGS="--pushRateLimit 1000" to your docker run command to specify a rate limit of 1000 pps for the proxy.

See the Wavefront Proxy configuration file for a full list.

Logging Customization for Containers

You can customize logging by mounting a customized log4j2.xml file. Here’s an example for Docker:

--mount type=bind, src=<absolute_path>/log4j2.xml, dst=/etc/wavefront/wavefront-proxy/log4j2.xml

See Logging above for additional background.

Installing Proxies on Multiple Linux Hosts

Ansible is an open-source automation engine that automates software provisioning, configuration and management, and application deployment. The Wavefront Ansible role installs and configures the Wavefront proxy, which allows you to automate Wavefront proxy installation on multiple Linux hosts.

For details, see the Setup tab in the Ansible built-in integration.

proxies