Varnish Configuration Language best practices: 3 VCL tips to put into practice

VCL, which stands for Varnish Configuration Language, is used to define your own caching policies in Varnish Cache and Varnish Enterprise. It is a real programming language whose syntax can in some cases be the same as the C language.

The upside of learning VCL is that, once you master it, you are able to define perfect caching policies and ensure that your Varnish Cache instance behaves exactly how you expect in every circumstance and under a variety of conditions.

The downside is, of course, that you will have to learn a new programming language.

In this blog post we won’t discuss the syntax itself, but most likely I’ll point out some interesting VCL tips and not very well-known VCL rules.

1. builtin.vcl is always appended to default.vcl
   When you install Varnish Cache or Varnish Enterprise, other than the binaries you also get two *.vcl files:
   
   builtin.vcl: Varnish Cache’s built-in behavior if you don’t define any other .vcl file. A built-in behavior is necessary to avoid wrong or even disruptive caching policies and to give you the ability to start caching immediately.
   
   default.vcl: this is the VCL file you can tweak to change the behavior of Varnish Cache.
   What is not very well-known is that builtin.vcl is, if no “return(action)” is defined, always appended to the default.vcl or any other VCL file you might have defined.
   
   For example:
   if in you default.vcl you define:

           sub vcl_hash {
     
   	}
       
   your Varnish Cache will instead see:
   

   	sub vcl_hash {
   	    hash_data(req.url);
   
       	if (req.http.host) {
   	       hash_data(req.http.host);
   	    } else {
   	       hash_data(server.ip);
   	    }
       	return (lookup);
   	}
   

   This is because the builtin subroutine for vcl_hash will be appended to your empty vcl_hash.

   As a rule of thumb, we strongly suggest that you let builtin.vcl be appended to your VCL to avoid unhappy situations.
   If you don’t want the builtin.vcl to be appended to your VCL file, then you need to specify a “return(action)”, which exits one state/subroutine and instructs Varnish to proceed to the next state/subroutine. If you must “return” from any subroutine, please test your VCL extensively before using it in a production environment. (More on Varnishtest).
   
   

2.  PCRE
   Varnish Cache and Varnish Enterprise rely on PCRE to process regular expressions.
   

     if (bereq.url ~ "\.(aif|aiff|au|avi|bin|bmp|cab|carb|cct|cdf|class|css)$"    ||
           bereq.url ~ "\.(dcr|doc|dtd|eps|exe|flv|gcf|gff|gif|grv|hdml|hqx)$"    ||
           bereq.url ~ "\.(ico|ini|jpeg|jpg|js|mov|mp3|nc|pct|pdf|png|ppc|pws)$"  ||
           bereq.url ~ "\.(swa|swf|tif|txt|vbs|w32|wav|wbmp|wml|wmlc|wmls|wmlsc)$"||
           bereq.url ~ "\.(eot|woff|svg|ttf)"||
           bereq.url ~ "\.(xml|xsd|xsl|zip)$")
     

   PCRE sometimes misbehaves and crashes when heavy recursive regular expressions are used. You can try to simplify as much as possible your regular expressions, breaking them down into smaller expressions.
   It is also suggested that you increase the thread_pool_stack parameter, which represents the thread stack size.
   
   

3. Strange hashing schemes:
   

   	sub vcl_hash {
   		hash_data(req.http.Cookie);
   		return(lookup);
   	}
       
   

   This VCL snippet inserts and searches for objects in cache based only on the request Cookie header and here we have at least two problems:
   

1. If the request has no Cookie header (very unlikely), no object will be inserted in cache. This is already in conflict with the purpose Varnish Cache serves, which is caching content and making your website faster. If no content is cached, then you can’t really see any performance improvements.
2. Cookie headers are unique to a user. Therefore when we insert objects in cache based on a Cookie header most likely the objects in cache will be equal one to another, but will be considered as different by Varnish Cache because the Cookie header changes based on the user. This leads to a very fragmented cache that contains more than a single copy of the same content.
   
   If you want to change the default Varnish Cache or Varnish Enterprise hashing scheme, do consider every possible scenario and their consequences.
   

Hopefully this blog post will help anyone who is an active VCL writer to define better and more secure caching policies.

If you want to learn more about VCL, the Varnish-Book is a great place to start.