When in doubt, test

I remember very well my first encounter with Varnish. It was this thing with a weird name that worked differently from the tools I had used before. Trying to understand why it was so unfamiliar quickly settled things and Varnish's architecture suddenly became an eye-opener.I dug in; I learned all about Varnish; I took on the challenge of introducing Varnish to people.

A couple of years later, it is less challenging and name recognition is significant enough that the lack of familiarity happens a lot less frequently. The people I meet tend to at least know what Varnish is. You might be wondering why I'm telling this story. The reason is quite simple: Varnish was once an unknown quantity and no longer is. And the same can be said for many of Varnish's benefits. In this case, a built-in test framework. It is a very little-known fact that a standard Varnish distribution comes with a test framework, and it is even less known how to use it.

Introducing Varnishtest

Starting with Varnish is not easy, especially when you're a developer not familiar with systems programming but still want to tinker with it. My user story from back then is:

As a newcomer to Varnish Cache,
I want to play with its source code,
Just for funzies and giggles.

I quickly started a VMOD, because that was the easiest way to write something, and a good starting point to explore the Varnish code base. And once you enter the VMOD realm, you realize that varnishtest exists. And then you start looking for documentation and realize there's none. The varnishtest manual says this:

The script language used for Varnishtest is not a strictly defined language. The best reference for writing scripts is the varnishtest program itself. In the Varnish source code repository, under bin/varnishtest/tests/, all the regression tests for Varnish are kept.

I followed suit, read the whole source code for Varnishtest and various test cases. Next thing I knew a blog post was born.

Retrospectively, when I read that piece now I realize how little I understood then. But Varnish is not so good at naming things and considering that this is one of the two hard problems in computer science, and Varnish having to solve the other one, it's a situation we can tolerate.

Varnishtest in itself is not that bad but for instance, there's a sema command, which stands for semaphore but it acts more like a barrier (spoiler alert, it may change). Little details can be confusing, and it becomes worse when they add up.

Y U NO TEST?

Tests are a means to increase confidence in the reliability of a piece of software, and yet I rarely see Varnish fit into the test infrastructure. Just like a backup is useless if it can't be restored, your cache is useless or can even become harmful if you can't refresh it at will. Testing your cache policy will give you confidence in your implementation, an automated test suite opens the door to agile (should I say mainstream?) practices like continuous integration.

The main reason for not testing is that Varnishtest is little known and people don't realize it's available by default. It is not surprising, and I have seen this pattern numerous times. Ask a Java developer to enumerate the tools that come with the JDK they are using on production (or even on their development box) and you'll be lucky if they can remember half of them without looking.

If you read these lines, you now know about Varnishtest, so I have fixed the main issue. There are other roadblocks. I already mentioned the lack of documentation of the test language. I'm planning to fix that for the next major release.

Another problem is the perception of Varnish as an infrastructure detail. The problem with a cache is that though it can be transparent to the user, it requires proper administration. What if you publish confidential and sensitive data by accident and need to remove them before they get indexed by search engines? Do you really want such an accident to become a nightmare because you didn't test?

That leads me to the last problem for realistic testing. Varnishtest is quite good at testing Varnish and the test suite achieves a decent coverage for a product that has almost no unit tests. What about testing Varnish in front of your backends?

Using Varnishtest

There are several incentives for learning to use Varnishtest. One is to check how Varnish behaves. When in doubt, it's often easy to prove or disprove a hypothesis, and it's always better than relying on folklore. I like to use it to illustrate articles or even support tickets.

VMOD writers tend to use Varnishtest, which is great. I don't remember a single VMOD I looked at that didn't have a proper test suite. Except maybe one, but some things are harder to test than others, and Varnishtest is mostly good at testing Varnish.

Of course this post was initially for general users of Varnish. It has always been possible to use Varnishtest to test your VCL, but it has not always been easy. It's even becoming easier as time passes, but there are things to consider. Since Varnish 4.1 it's even easier to test Varnish against your backend. I'm tempted to write a post on how to actually test your VCL, possibly with real backends, leave a comment and let me know if you are interested.

varnishtest information is included throughout The Varnish Book, which you can download to get started.