Proxy protocol to get the original client IP identity in Varnish 4.1

HAProxy’s proxy protocol has been supported as part of Varnish 4.1 for a few months now. The proxy protocol's main purpose is to transport client connection information from the originating source requesting the connection. It adds a small header about the origin connection.

As we know Varnish does not support TLS/SSL natively so in order to terminate those connections we need a TLS terminator. Although you can solve this with a hardware load balancer (F5 BigIP and such), we recommend you use a standalone piece of software, such as HAProxy, Pound or our own Hitch (previously Stud) for this.

In this post we will use Hitch. Made by Varnish Software, Hitch is a standalone open source project used widely for TLS/SSL termination. It sits in front of Varnish and makes sure the content is delivered over https to the client while passing the data untouched through the wire.

The requests Varnish sees come from Hitch, so if you need the client’s IP address, you get the IP address of the Hitch proxy that sits in front, not the actual client IP address. So Varnish does not know if the IP address in the X-Forwarded-For (XFF) header is the user’s. Hitch is a dumb proxy so it cannot add an XFF header. It just passes bytes and is obvious to the data transferred, ie not an HTTP proxy. Varnish always adds an XFF header based on the client.ip variable. The TLS traffic would thus gets a wrong XFF header based on Hitch <-> varnish connection rather than the client   <-> Hitch connection.

With the proxy protocol supported, Varnish instantly knows what the original IP was, as that information will be passed on as well. Proxy protocol allows Hitch to send a short header just before the main connection data on the hitch backend connection. Varnish will populate the client.ip(/port) and server.ip(/port) based on the preamble sent by hitch to varnish when the connection is set up. Thus the generated XFF header will now show what you would expect regardless of if the connection was TLS and went through Hitch or was unencrypted and went straight to Varnish.

When we enable Varnish by default we bind it to port 80. In order to make Varnish aware of the Hitch proxy protocol we need to instruct Varnish to listen on a different port using this protocol. So we need to add an additional parameter and make it listen on the extra port. Thus we have a primary port, which will be http in Varnish, and we add a secondary port, which we make listen to the proxy protocol. Then we can make sure that the client.ip will be set to the real client IP supplied in the protocol itself - not the SSL/TLS terminating proxy.

An example is given below to show how to edit the Varnish command line where http traffic is on port 80 and proxy traffic is handled on port 6081. The ­-a startup argument syntax has been expanded to allow for this:

varnishd -f /etc/varnish/default.vcl -a :80 -a 127.0.0.1:6081,PROXY

Configure Hitch

After installing Hitch, we need to have our own valid certificate, and we can use it to set up Hitch. The certificate is expected in pem format. In hitch.conf (/etc/hitch/hitch.conf), you need to have the following setup and then start Hitch with the new configuration:

# Listening
frontend = "[*]:443"
ciphers  = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"

# Send traffic to the Varnish backend using the PROXY protocol
backend        = "[::1]:6081"
write-proxy-v2 = on

# List of PEM files, each with key, certificates and dhparams
pem-file = "/etc/hitch/testcert.pem"

With Varnish 4.1 supporting the proxy protocol VCL will have four different IP vars (client.ip, server.ip, remote.ip. local.ip). By default (with no proxy proto) client.ip and server.ip will be set to the values of remote.ip and client.ip respectively. With proxy protocol they will get their values set from the proxy header.

You can learn a lot more about Hitch from its documentation. Take a look.

Photo (c) 2009 Andrea Castelli used under Creative Commons license.