Advanced Proxy Configuration

Even without additional customization the Wavefront proxy ingests metrics and forwards them to the Wavefront service in a secure, fast, and reliable manner. If needed, you can customize your proxy.

• Proxy configuration properties allow you to changing how the proxy processes your data. For example, you can change ports or perform other advanced installation management.
• Proxy preprocessor rules allow you to manipulate incoming metrics before they reach the proxy, for example, you could remove confidential text strings or replace unacceptable characters.
• Log files can help in case of problems.

Proxy File Paths

By default, proxy files are installed in the following locations.

• Configuration properties: <wavefront_config_path>
  • Linux - /etc/wavefront/wavefront-proxy
  • Mac - /usr/local/etc/wavefront/wavefront-proxy
  • Windows - C:\Program Files (x86)\Wavefront\conf
• Parse log data: <wavefront_log_path>
  • Linux - /var/log/wavefront
  • Mac - /usr/local/var/log/wavefront
  • Windows - C:\Program Files (x86)\Wavefront
• <wavefront_spool_path>
  • Linux - /var/spool/wavefront-proxy
  • Mac - /usr/local/var/spool/wavefront-proxy
  • Windows - C:\Program Files (x86)\Wavefront\bin

Data Buffering

If the Wavefront proxy is unable to post received data to the Wavefront servers, it buffers the data to disk across a number of buffer files, and then tries to resend the points once the connection to the Wavefront servers is available again. If this buffering occurs, you’ll see lines like this in wavefront.log:

2013-11-18 18:02:35,061 WARN  [com.wavefront.daemon.QueuedSshDaemonService] current retry queue sizes: [1/0/0/0]

By default, there are 4 threads (and 4 buffer files) waiting to retry points once the connections are up; this line shows how many blocks of points have been stored by each thread (in this case, the first thread has 1 block of queued points, while the second, third, and fourth threads all have 0 blocks). These lines are only printed when there are points in the queue; you’ll never see a line with all 0’s in the queue sizes. Once the connection to the Wavefront servers has been established, and all the threads have sent the past data to us, you’ll see a single line like this in wavefront.log:

2013-11-18 18:59:46,665 WARN [com.wavefront.daemon.QueuedSshDaemonService] retry queue has been cleared

Logging

The Wavefront proxy supports two log files: proxy log and blocked point log.

To keep the log file sizes reasonable and avoid filling up the disk with logs, both log files are automatically rotated and purged periodically. Configure the log file locations and rotation rules in <wavefront_config_path>/log4j2.xml. For details on log4j2 configuration, see Log4j Configuration.

If you’re using proxies in containers, you can mount the log files, as discussed below.

Proxy Log

By default, proxy log entries are logged to <wavefront_log_path>/wavefront.log. The log file is rolled over every day and when its size reaches 100MB. When there are 31 log files, older files are deleted.

If you want to set logs for Jaeger and Zipkin integrations, see Logging for Jaeger and Zipkin.

Blocked Data Log

You can log all the raw blocked data separately or log different entities into their separate log files.

• Log the block data separately
  Follow these steps:
  1. Open the <wavefront_config_path>/log4j2.xml configuration file.
  2. To log all the block data, uncomment the corresponding section.

      <AsyncLogger name="RawBlockPoints" level="WARN" additivity="false">
         <AppenderRef ref="BlockedPointsFile" />
      </AsyncLogger>
     

     By default, blocked point entries are logged to the <wavefront_log_path>/wavefront-blocked-points.log file and the log file is rolled over every day when its size reaches 100MB. When there are 31 log files, older files are deleted. You can customize the configurations to suit your environment.

• Set up separate log files for blocked entities
  Follow these steps:
  1. Uncomment or add the configurations under Appenders and Loggers in the <wavefront_config_path>/log4j2.xml configuration file.

        <!-- Log the blocked histograms. If you don't need a separate log file for it,
        don't add this configuration to the file.-->
        <AsyncLogger name="RawBlockedHistograms" level="WARN" additivity="false">
           <AppenderRef ref="[Enter_Your_File_Name]"/>
       </AsyncLogger>
     
       <!-- Logs the blocked points for spans. If you don't need a separate log file for it,
       don't add this configuration to the file.-->
       <AsyncLogger name="RawBlockedSpans" level="WARN" additivity="false">
           <AppenderRef ref="[Enter_Your_File_Name]"/>
       </AsyncLogger>
     
       <AsyncLogger name="RawBlockPoints" level="WARN" ADDITIVITY="FALSE"/>
     	   <AppenderRef ref=”BlockedPointsFile”/>
       </AsyncLogger>
     

  2. Add the names of the block points, which you uncommented in the log4j2.xml file, to the <wavefront_config_path>/wavefront.conf file.
     Example:

        blockedPointsLoggerName = RawBlockedPoints
     
        # Add this if you added the appender for histograms in the log4j2.xml file.
        blockedHistogramsLoggerName = RawBlockedHistograms (RawBlockedPoints by default)
     
        # Add this if you added the appender for spans in the log4j2.xml file.
        blockedSpansLoggerName = RawBlockedSpans (RawBlockedPoints by default)
     

  Tip: You must update both the <wavefront_log_path>/log4j2.xml file and the <wavefront_config_path>/wavefront.conf file to get separate log files for blocked entities.

Configuration Properties

This section gives details on proxy configuration properties. All properties are also listed, in the wavefront.conf.default file on Github.

General Configuration Properties

Authentication Proxy Properties

Because the proxy is running in your local network by default, communication to the proxy is un-authenticated. If you want to authenticate inbound traffic to the proxy, use the settings in this section.

The Wavefront proxy must authenticate outbound traffic to the Wavefront service. See Authenticate Incoming HTTP Requests at the Proxy for step-by-step instructions.

Tracing Proxy Properties

Histogram Configuration Properties

Wavefront supports additional histogram configuration properties, shown in the following table. Note the requirements on the state directory and the effect of the two persist properties listed at the bottom of the table.

Configure a Proxy in a Docker Container

You can use the in-product Docker with cAdvisor or Kubernetes integration if you want to set up a proxy in a container. You can then customize that proxy.

Proxy Versions for Containers

For containers, the proxy image version is determined by the image property in the configuration file. You can set this to image: wavefronthq/proxy:latest, or specify a proxy version explicitly. The proxies are not stateful. Your configuration is managed in your yaml file. It’s safe to use proxy:latest – we ensure that proxies are backward compatible.

Restrict Memory Usage for a Docker Container

To restrict memory usage of the container using Docker, you need to add a JAVA_HEAP_USAGE environment variable and restrict memory using the -m or --memory options for the docker run command. The container memory contraint should be at least 350mb larger than the JAVA_HEAP_USAGE environment variable.

To restrict a container’s memory usage to 2g with Docker run:

docker run -d --name wavefront-proxy ... -e JAVA_HEAP_USAGE="1650m" -m 2g ...

To limit memory usage of the container in Kubernetes use the resources.limits.memory property of a container definition. See the Kubernetes doc.

Customize Proxy Settings for Docker

When you run a Wavefront proxy inside a Docker container, you can tweak proxy configuration settings that are properties in the wavefront.conf file directly from the Docker run command. You use the WAVEFRONT_PROXY_ARGS environment variable and pass in the property name as a long form argument, preceded by --.

For example, add -e WAVEFRONT_PROXY_ARGS="--pushRateLimit 1000" to your docker run command to specify a rate limit of 1000 pps for the proxy.

See the Wavefront Proxy configuration file for a full list.

Log Customization for Docker Containers

You can customize logging by mounting a customized log4j2.xml file. Here’s an example for Docker:

--mount type=bind, src=<absolute_path>/log4j2.xml, dst=/etc/wavefront/wavefront-proxy/log4j2.xml

See Logging above for additional background.

Authenticate Incoming HTTP Requests at the Proxy

The Wavefront proxy can be configured to authenticate inbound traffic. Below are the steps for setting up the different supported authentication methods.

Authentication Method Static Token


Required properties: authStaticToken


In order for data sent via HTTP to the proxy to be accepted, the request will need to include a token matching whatever is configured for authStaticToken. The token can be specified in the header or in the query string.

Authentication Method HTTP_Get or OAUTH2



Required properties: authTokenIntrospectionServiceUrl


Optional properties: authTokenIntrospectionAuthorizationHeader, authResponseRefreshInterval, authResponseMaxTtl


If you have a service that can validate tokens, use this approach.

• Use OAUTH2 if your Introspection Service is RFC7662 compliant.
• Otherwise, use HTTP_GET.

For either method, the service must return a 2xx code for valid tokens. According to the standard, OAUTH2-compliant services return a JSON object that contains the active field.