Two-Minute Tech Tuesdays - Built-in VCL

This week’s episode of Two Minute Tech Tuesday explains how the built-in Varnish Configuration Language (VCL) works. The built-in VCL is VCL code that is executed by default, even if you don't write any VCL yourself.

The built-in VCL is executed behind the scenes as requests are received. It is possible to change the behavior by writing your own VCL, however, it all revolves around the execution of a finite state machine.

The finite state machine in the diagram below features a collection of states; the VCL subroutines containing the built-in VCL code.

There are also some actions involved that allow transitions from one state to the other. Here's an example of some built-in VCL code or the vcl_recv subroutine that is executed when a request is received. 

sub vcl_recv {
    if (req.method == "PRI") {
       /* This will never happen in properly formed traffic (see: RFC7540) */
       return (synth(405));
    }
    if (!req.http.host &&
      req.esi_level == 0 &&
      req.proto ~ "^(?i)HTTP/1.1") {
        /* In HTTP/1.1, Host is required. */
        return (synth(400));
    }
    if (req.method != "GET" &&
      req.method != "HEAD" &&
      req.method != "PUT" &&
      req.method != "POST" &&
      req.method != "TRACE" &&
      req.method != "OPTIONS" &&
      req.method != "DELETE" &&
      req.method != "PATCH") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }

It will return synthetic errors when preconditions aren't met and it will also downgrade the connection from an HTTP connection to a TCP connection when it believes the content is no longer HTTP because of an invalid request method.

It can also bypass the cache when the request either uses an uncacheable request method or when the request contains an Authorization or a Cookie header. Under all other circumstances, the cache lookup is made by creating a hash and a transition is made to the vcl_hash subroutine.

Inside vcl_hash, a hash key is created by calling the hash_data() function and using both the URL and the Host header. This hash is used for a cache lookup. When the cache lookup succeeds, we transition to vcl_hit.

Inside vcl_hit, content will be delivered to the client if the object is not expired. If it is expired but has some grace time left, we will deliver the stale version while doing a background fetch. Otherwise, we're dealing with an expired and out of grace object and we will transition to vcl_miss.

Another crucial subroutine is the vcl_backend_response subroutine. The vcl_backend_response subroutine is reached after a backend fetch takes place and the server responds just before the object is stored in the cache.

When the request that triggers it is deemed uncacheable, we directly deliver the content without storing it in the cache. We can also mark responses as uncacheable for future requests. We do this if preconditions aren't met, such as TTLs that are lower than or equal to zero, set-Cookie headers being set, or Cache-Control headers implying that the object cannot be cached.

The built-in VCL is a crucial concept. If you want to comfortably write VCL, you must understand the Varnish finite state machine with its states and transitions. You must also know the built-in VCL in each of these subroutines. If you want to learn more, check out our Developer Portal.