Caching partial objects with varnish 4.0

HTTP allows a client to ask for only a part of an object. This is often called a "range request", because the header involved is called Range.

Varnish 4.0 will by default go to the backend and fetch the entire object, and when the part that the client asked for arrives, serve that as fast as it comes in.

There are however times when you don't want the entire object to be fetched from the backend server. Let us say this is a 4GB combined video file, and the client only asked for a few kilobytes of it.

In these cases it might be a better solution just to cache the individual chunks of the file, typically what a client would ask for. This is especially true for video streaming where the clients would ask for the same ranges/sections of the file. In such cases, caching partial objects can be useful.

This works a bit against the normal way Varnish wants things to work, so we have to add some VCL 

    sub vcl_recv {
        if (req.http.Range ~ "bytes=") {
            set req.http.x-range = req.http.Range;
        }
    }

    sub vcl_hash {
        if (req.http.x-range ~ "bytes=") {
                hash_data(req.http.x-range);
                unset req.http.Range;
        }
    }

    sub vcl_backend_fetch {
        if (bereq.http.x-range) {
            set bereq.http.Range = bereq.http.x-range;
        }
    }

    sub vcl_backend_response {
        if (bereq.http.x-range ~ "bytes=" && beresp.status == 206) {
            set beresp.ttl = 10m;
            set beresp.http.CR = beresp.http.content-range;
        }
    }

    sub vcl_deliver {
        if (resp.http.CR) {
            set resp.http.Content-Range = resp.http.CR;
            unset resp.http.CR;
        }
    }

I'll go through each part to make it clear what happens:

1) First we take a copy of the Range header to a separate X-Range. This isn't strictly necessary, but it is nice to be able to comment this out so you can test the behavior manually by adding X-Range with curl.

2) By adding the Range header to the object hash, we get different lookups for each range. Overlapping ranges will not be detected or coalesced by this, so beware if clients pick their ranges wild.

3) Since we use a different header internally, we need to set it back to Range on the backend request. This will tell the backend server that we need just a chunk, not the whole file.

4) If we do get a partial response back from the backend server (206 response code), we store the Content-Range header we got. Internally Varnish will filter this from being stored under its original name, so we need to use a different header name in the stored object.

5) Finally we're ready to deliver the response to the client, and we put the Content-Range header back where it belongs.

After loading this VCL, any Range requests sent by the client will have Varnish fetch just the segment asked for, and when the next client comes asking for the same segment, we have it in cache.