Advanced Proxy Configuration and Installation

You can configure proxies using a configuration file, and you can perform advanced installation management such as installing proxies in a container.

In addition to the proxy configuration properties discussed here you can also use proxy preprocessor rules. These rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.

Proxy Configuration Properties

The main Wavefront proxy configuration file is maintained in <wavefront_config_path>/wavefront.conf (<wf_config_path>/wavefront.conf). The configuration file offers many options for changing how the proxy processes your data. There are optional configuration files for rewriting metrics and parsing log data. The default values work well in many cases, but you can adjust them as needed. After changing a configuration option, restart the proxy service.

Paths

In this section, file paths use the following conventions and values:

• <wavefront_config_path>
  • Linux - /etc/wavefront/wavefront-proxy
  • Mac - /usr/local/etc/wavefront/wavefront-proxy
  • Windows - C:\Program Files (x86)\Wavefront\conf
• <wavefront_log_path>
  • Linux - /var/log/wavefront
  • Mac - /usr/local/var/log/wavefront
  • Windows - C:\Program Files (x86)\Wavefront
• <wavefront_spool_path>
  • Linux - /var/spool/wavefront-proxy
  • Mac - /usr/local/var/spool/wavefront-proxy
  • Windows - C:\Program Files (x86)\Wavefront\bin

General Proxy Properties and Examples

This section lists:

• General proxy configuration properties
• Metrics proxy configuration properties
• Tracing proxy configuration properties

See the Histogram Configuration Properties table below for properties specific to histogram distributions.

Tracing Proxy Properties and Examples

Histogram Configuration Properties

Wavefront supports additional histogram configuration properties, shown in the following table. Note the requirements on the state directory and the effect of the two persist properties listed at the bottom of the table.

Data Buffering

If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:

2013-11-18 18:02:35,061 WARN  [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]

By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:

2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared

Logging

The Wavefront proxy supports two log files: proxy log and blocked point log.

To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. Configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.

If you’re using proxies in containers, you can mount the log files, as discussed below.

Proxy Log

By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.

If you want to set logs for Jaeger and Zipkin integrations, see Logging for Jaeger and Zipkin.

Blocked Data Log

You can log all the raw blocked data separately or log different entities into their separate log files.

• Log the block data separately
  Follow these steps:
  1. Open the <wavefront_config_path>/log4j2.xml configuration file.
  2. To log all the block data, uncomment the corresponding section.

      <AsyncLogger name="RawBlockPoints" level="WARN" additivity="false">
         <AppenderRef ref="BlockedPointsFile" />
      </AsyncLogger>
     

     By default, blocked point entries are logged to the <wavefront_log_path>/wavefront-blocked-points.log file and the log file is rolled over every day when its size reaches 100MB. When there are 31 log files, older files are deleted. You can customize the configurations to suit your environment.

• Set up separate log files for blocked entities
  Follow these steps:
  1. Uncomment or add the configurations under Appenders and Loggers in the <wavefront_config_path>/log4j2.xml configuration file.

        <!-- Log the blocked histograms. If you don't need a separate log file for it,
        don't add this configuration to the file.-->
        <AsyncLogger name="RawBlockedHistograms" level="WARN" additivity="false">
           <AppenderRef ref="[Enter_Your_File_Name]"/>
       </AsyncLogger>
     
       <!-- Logs the blocked points for spans. If you don't need a separate log file for it,
       don't add this configuration to the file.-->
       <AsyncLogger name="RawBlockedSpans" level="WARN" additivity="false">
           <AppenderRef ref="[Enter_Your_File_Name]"/>
       </AsyncLogger>
     
       <AsyncLogger name="RawBlockPoints" level="WARN" ADDITIVITY="FALSE"/>
     	   <AppenderRef ref=”BlockedPointsFile”/>
       </AsyncLogger>
     

  2. Add the names of the block points, which you uncommented in the log4j2.xml file, to the <wavefront_config_path>/wavefront.conf file.
     Example:

        blockedPointsLoggerName = RawBlockedPoints
     
        # Add this if you added the appender for histograms in the log4j2.xml file.
        blockedHistogramsLoggerName = RawBlockedHistograms (RawBlockedPoints by default)
     
        # Add this if you added the appender for spans in the log4j2.xml file.
        blockedSpansLoggerName = RawBlockedSpans (RawBlockedPoints by default)
     

  Warning: You must update the <wavefront_log_path>/log4j2.xml file and the <wavefront_config_path>/wavefront.conf file to get separate log files for blocked entities.

Configuring a Proxy in a Container

You can use the in-product Docker with cAdvisor or Kubernetes integration if you want to set up a proxy in a container. You can then customize that proxy.

Proxy Versions for Containers

For containers, the proxy image version is determined by the image property in the configuration file. You can set this to image: wavefronthq/proxy:latest, or specify a proxy version explicitly. The proxies are not stateful. Your configuration is managed in your yaml file. It’s safe to use proxy:latest – we ensure that proxies are backward compatible.

Restricting Memory Usage for the Container

To restrict memory usage of the container using Docker, you need to add a JAVA_HEAP_USAGE environment variable and restrict memory using the -m or --memory options for the docker run command. The container memory contraint should be at least 350mb larger than the JAVA_HEAP_USAGE environment variable.

To restrict a container’s memory usage to 2g with Docker run:

docker run -d --name wavefront-proxy ... -e JAVA_HEAP_USAGE="1650m" -m 2g ...

To limit memory usage of the container in Kubernetes use the resources.limits.memory property of a container definition. See the Kubernetes doc.

Customizing Proxy Settings for Docker

When you run a Wavefront proxy inside a Docker container, you can tweak proxy configuration settings that are properties in the wavefront.conf file directly from the Docker run command. You use the WAVEFRONT_PROXY_ARGS environment variable and pass in the property name as a long form argument, preceded by --.

For example, add -e WAVEFRONT_PROXY_ARGS="--pushRateLimit 1000" to your docker run command to specify a rate limit of 1000 pps for the proxy.

See the Wavefront Proxy configuration file for a full list.

Logging Customization for Containers

You can customize logging by mounting a customized log4j2.xml file. Here’s an example for Docker:

--mount type=bind, src=<absolute_path>/log4j2.xml, dst=/etc/wavefront/wavefront-proxy/log4j2.xml

See Logging above for additional background.

Installing Proxies on Multiple Linux Hosts

Ansible is an open-source automation engine that automates software provisioning, configuration and management, and application deployment. The Wavefront Ansible role installs and configures the Wavefront proxy, which allows you to automate Wavefront proxy installation on multiple Linux hosts.

For details, see the Setup tab in the Ansible built-in integration.