Varnish Shared Memory Log errors and solutions

The Varnish distribution comes with several utility programs. The main one is the Varnish daemon varnishd. It accepts and handles HTTP requests from clients and it’s responsible for the cache itself. Among the binaries you get from your Varnish installation, there are five programs that read the Varnish Shared Memory Log (VSL), parse it or extract data from it and present it to you: varnishlog, varnishstat, varnishncsa, varnishtop and varnishhist.

In this blog post I’ll throw the possible errors you can encounter when launching one of the above mentioned programs. You might ask… why? A few weeks back I spent a lot of time trying to fix my Varnish installation, and I fell into the install-remove varnish loop in an attempt to fix it. When I realized a magic fix was never going to happen, I went through the source code and eventually fixed my Varnish.

(I could have simply destroyed my virtual machine, but I really wanted to figure out why my Varnish wasn’t happy.)

VSL or SHMLOG is a circular buffer; therefore, it is not persistent (mount on a temporary file storage to make it persistent). By default it’s 80MG large and about half of it is used by Varnish to log transaction-related info and the other half to write counters.

Every time you run one of the programs using the VSL, the program tries to read a VSL chunk to extract the relevant data and present them to you. If the operation fails you can get one of the following errors:

• No VSL chunk found (child not started?): This error is self-explanatory, most probably your Varnish hasn’t started properly. You can check the syslog to understand why Varnish is not running.
  If your Varnish is running and you still get this error, it could be that the program is not able to find the “_.vsm” file, a “locate _.vsm” should help you to find the file and fix your installation.
  In most cases a restart of Varnish fixes the issue.

• Out of memory: Varnish is not able to allocate any memory for one of the structures used to handle the VSM. If you have a look at the memory situation of your server, you will be able to figure out which is the best solution for you.

• Cursor initialization failure: This error happens when varnishd is writing to the Shared Memory Log faster then the program that is trying to read from it. Since the VSL is a circular buffer Varnish can overrun it and if it happens, the program reading the VSL will end up with inconsistent data. To be honest I’ve never seen anyone affected by this issue, but the solution would be to have a persistent VSL so the slow program accessing it will have enough time to read it with consistency.

• Not a VSL chunk: This is the error that I ended up spending a lot of time on. It’s quite hard to understand what’s going on with your Varnish even if you read the source code.
  Varnish is not able to read the VSL chunk simply because it can’t recognize that piece of buffer as a VSL chunk.

I faced this error on a virtual machine I use for quick testing at work, which means that throughout the day I install/remove different versions of Varnish either from packages or from source. This continually changing Varnish version created a situation where I had a specific Varnish version (4.0)  installed and a different libvarnishapi1 (4.1).

The fix for this error is pretty straightforward: make sure there are no leftovers from a previous installation. 
Running “ldd `which varnishlog`” will output the shared library dependencies for varnishlog:

At this point you should be able to check which Varnish and libvarnishapi versions you have   installed. Remove the wrong libvarnishapi version and install Varnish again to fix the mismatch.

Here I spent some time figuring out where I went wrong, but you can learn more about step-by-step caching with Varnish in our on-demand webinar.