The essential vmods all Varnish users should know about

Approximately two years ago we released Varnish Cache 3.0. In my view the most important feature was the support for modules. Thanks to the hello world vmod, which almost all of the modules out there are based on, it’s rather easy to write a trivial module. It is obviously more complicated to write more complex modules and since there is no sandboxing involved we see modules wrecking havoc in varnishd regularly. Memory leaks and segfaults are found in modules from time to time.

However, there are modules that can add really wonderful functionality without destabilizing your varnishd. In my opinion these are some of the modules all Varnish users should know about.

Variables in VCL
While we’ve been using req.http.foo and req.http.bar as variables since Varnish 1.0 it isn’t really ideal. Tollefs variable vmod fixes this. It adds associative arrays, so you can reference a variable using any arbitrary string, even ones you’ve got from the client.

var.set(req.http.user-agent, “some silly string”);

In addition the variable vmod also has global variables. This gives you the option to parameterize your VCL and then change its parameters dynamically. This allows you to do all sorts of nifty stuff such as switching from normal behaviour to a “under siege” configuration by calling a specific URL which would instantly switch timeouts, TTLs and other parameters.

Sorting of query string parameters
For Varnish the query string is opaque. It is input as a string into the hash function and it returns a pointer to a set of candidates for delivery. So, two almost similar URLs will return completely different hash keys although the content is the same.

• http://foo.com/index.html?p=5&q=3
• http://foo.com/index.html?q=3&p=5

If you sort the query string parameters you’ll normalize these URLs so they will yield the same hash key and therefore the same content. It’s a very simple way to boost your hit rate if you have lots of different apps generating links without enforcing a specific order to parameters.

There are myriad different query string sorting VMODs. Our favorite is boltsort, written as a result of a vmod hackathon at Vimeo. Just add

set req.url = boltsort.sort(req.url);

in vcl_recv and Bob’s your uncle.

Cookie manipulation in VCL
Cookies are kind of cumbersome in VCL. Operations such as “remove every cookie but this one” can be done through regular expressions but it’s difficult. Lasse picked up the challenge and has just published his cookie VMOD.

cookie.filter_except("userid,prefs");

This line removes all cookies except the userid and prefs cookie. Doing this with regular expressions is certainly possible but it looks pretty horrible.

Please note that the cookie VMOD is still rather new and I haven’t seen it in production yet.

Using ACLs with X-Forwarded-For - ipcast
Quite often there is a load balancer installed in front of Varnish and the way Varnish gets the IP address from the client is through the X-Forwarded-For. In order to make the ACLs work we need to convert the string to a native IP address. This is exactly what the ipcast vmod does. Just call

ipcast.clientip(req.http.x-forwarded-for);

and it will set client.ip to the address of the client.

Do you agree with me on this listing or are there other essential VMODs that you feel should be mentioned here?

If you're curious to learn more about VMODs and all about Varnish, download the Varnish Book.