/VS-logo-2020-197x60.png?width=136&height=60&name=VS-logo-2020-197x60.png "Varnish Software"){kind=logo}
When talking about caching, nearly all the attention goes to ensuring that content is properly stored in the cache. While that makes total sense from a performance perspective, people tend to forget that removing objects from the cache is equally as important.
Take for example a breaking news article: if the homepage of that news website is cached, the breaking news will not appear until the homepage is removed from the cache.
And when does that happen?
- When the Time To Live of that object in the cache has expired
- If the object is forcefully removed from the cache when the cache is full
- When the content management system of the news website explicitly sends a purge request to Varnish to remove the relevant objects from the cache
In this blog post, we’ll talk about the latter. Varnish has a broad variety of mechanisms to remove objects from the cache, ranging from purges to bans, to secondary key invalidations.
Let’s have a look at the individual methods.
Native purging in Varnish
The most common way to invalidate objects from the cache is through Varnish’s native purging mechanism. It requires modifying your VCL code, capturing the purge request and performing the purge.
Here’s some very basic example code that does just that:
vcl 4.1;
sub vcl_recv {
if (req.method == "PURGE") {
return (purge);
}
}
By hooking into the “vcl_recv” subroutine, we can override the behavior for incoming requests. We specifically capture requests that use the “PURGE” request method (instead of a typical “GET” or “POST” request that would otherwise be used).
When we receive the incoming PURGE request we perform a “return(purge)” call in Varnish which will remove the relevant object from the cache.
Objects are identified by their URL and the Host header. So by simply accessing the full URL with a “PURGE” request method, Varnish knows exactly what to do.
Here’s an example of a RAW HTTP request that performs the purge call:
PURGE /products HTTP/1.1
Host: example.com
And here’s the corresponding HTTP response:
HTTP/1.1 200 Purged
Server: Varnish
X-Varnish: 32770
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 240
Accept-Ranges: bytes
Connection: keep-alive
<!DOCTYPE html>
<html>
<head>
<title>200 Purged</title>
</head>
<body>
<h1>Error 200 Purged</h1>
<p>Purged</p>
<h3>Guru Meditation:</h3>
<p>XID: 32770</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
This response is a synthetic response: it didn’t come from the origin server and was created by Varnish itself. The typical output uses an HTML template that parses a couple of variables:
- The status code: “200” in our case
- The reason phrase: “Purged” in our case
- The Varnish transaction ID of the request: “32770” in our case
Don’t worry about the “Error” keyword appearing in the output, it’s just the basic template that you can modify in the “vcl_synth” subroutine. In a lot of cases, synthetic output is used to return errors.
Here’s the standard VCL code for the “vcl_synth” subroutine:
sub vcl_synth {
set resp.http.Content-Type = "text/html; charset=utf-8";
set resp.http.Retry-After = "5";
set resp.body = {"<!DOCTYPE html>
<html>
<head>
<title>"} + resp.status + " " + resp.reason + {"</title>
</head>
<body>
<h1>Error "} + resp.status + " " + resp.reason + {"</h1>
<p>"} + resp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>XID: "} + req.xid + {"</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
"};
return (deliver);
}
Securing your purge interface
Because purge requests are sent over the same interface as regular HTTP requests, there is a security risk: people with malicious intent could send unrestricted purge requests to Varnish, which could ultimately destabilize the entire platform because of a low hit rate.
That’s why it’s important to secure the purge interface. We typically do this through access control lists, but you can just as well require authentication.
Here’s the VCL code to protect your purges:
vcl 4.1;
acl purge {
"localhost";
"192.168.1.5";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405, "Not Allowed"));
}
return (purge);
}
}
The ACL is called “purge” and contains hostnames, IP addresses and IP subnets. Within the purge logic, we define an extra if-statement that checks whether the client’s IP address matches the ACL. If that’s not the case, we return a synthetic response with a “405” status code and a “Not Allowed” reason phrase.
How to perform purges?
In an earlier paragraph, I showed you the HTTP request below:
PURGE /products HTTP/1.1
Host: example.com
This is a raw HTTP request that allows you to purge http(s)://example.com/products. While it is syntactically correct, it’s not a turn-key solution.
If you’re doing purges on the command line, you could use cURL and end up with the following command:
curl -XPURGE https://example.com/products
The “-X” option is there to set the request method, which results in “-XPURGE” if you want to perform a purge.
If you’re planning to integrate the purge interface in your application code, your language or development framework probably has a programmable HTTP client or a cURL integration.
If you use one of the popular content management systems, there’s probably a Varnish plugin available that handles the purges for you. Have a look at the Developer Portal tutorials if you need some inspiration.
The need for soft purges
Besides simplicity and easy implementation, one of the main advantages of Varnish’s native purging capabilities is the fact that it removes the object from the cache immediately.
Once the synthetic response is returned, you know the purge has finished. But this very asset can also be considered a liability. Especially when the application is slow out of the gate.
When an object is removed from the cache, the burden is put on the next client to wait until the origin server has responded before it can receive the content. If that origin server is slow, the quality of experience for the client will deteriorate. If multiple people are trying to access that content, they all have to wait, despite our request collapsing capabilities.
What if we could leverage Varnish’s built-in “stale while revalidate'' features to serve stale content while a newer version of the content is asynchronously fetched. The end result is the same (updated content is loaded), but the end-user inconvenience is a lot less severe.
Thanks to the “purge” VMOD, a separate module shipped with Varnish, we can issue soft purges. Here’s the VCL code:
vcl 4.1;
import purge;
sub vcl_recv {
if (req.method == "PURGE") {
return (hash);
}
}
sub vcl_hit {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_miss {
if (req.method == "PURGE") {
call my_purge;
}
}
sub my_purge {
set req.http.purged = purge.soft(0s,30s,120s);
if (req.http.purged == "0") {
return (synth(404));
} else {
return (synth(200, "Purged"));
}
}
While the interface for the consumer of the purge endpoint is nearly identical, the inner workings are a bit different. The “purge.soft()” function can only be called within the “vcl_hit” or “vcl_miss” subroutines. To avoid too much code duplication we wrote a custom “my_purge” subroutine that is called by “vcl_hit” and “vcl_miss”.
We’re calling “purge.soft()” with a zero TTL, to mark that the object has expired. We set the grace value to 30 seconds, which indicates that stale objects can be served up to 30 seconds past the expiration time of the object. Meanwhile, Varnish will asynchronously fetch the updated version.
The keep time is set to 120 seconds and it’s there to keep the object around. While expired and out-of-grace objects will not be served from the cache, keeping them around still offers the possibility of “conditional revalidation”. This means that entity tags that might have been set in the original HTTP response can be sent as “If-None-Match” headers by both the client and Varnish.
If the entity tag that was stored on the origin matches the value of the “If-None-Match” header, the content hasn’t changed and a “304 Not Modified” response can be returned without sending any payload over the wire. This has the potential of being a lot faster than a regular “200 OK” response, not just because there’s no data being sent over the wire, but because it can result in a faster time to first byte if the origin server was optimized for conditional requests.
Long story short: for slow applications, soft purges soften the blow and ensure an unlucky end-user doesn’t have to wait for the slow backend to respond.
Purge limitations
Varnish’s native purges are pretty simple, they are easy to implement and thanks to the purge VMOD, we can even perform soft purges. We already discussed the pros and cons of the immediate removal of objects from the cache, but there is one major drawback we need to discuss.
Purges can only remove a single object from the cache in a single call. If one content block ends up on a variety of web pages, every single one of them needs to be invalidated. Sending individual purges for all these pages is not really efficient.
Varnish offers “bans” and “secondary keys” as a way to counter this limitation.
Bans
Bans are another native Varnish feature that invalidates objects based on so-called “ban expression”. These expressions have the ability to match multiple objects, which makes banning a good alternative to purges.
Here’s an example of a ban expression:
obj.status > 200
This expression will match every object that has a response status code greater than 200.
There are plenty of operators you can use and you can even chain multiple expressions with logical “and” operators, “or” operators, and negations. Here’s a different example that uses some of these operators:
req.http.host == example.com && req.url ~ ^/admin
This example matches all objects that have “example.com” as their host header and whose URL starts with “/admin”. The tilde operator is used to match regular expressions, hence the “^/admin” regex.
Adding expressions to the ban list can be done using the “varnishadm ban” subcommand. Here’s an example:
varnishadm ban "req.url ~ ^/admin && req.http.host == example.com"
And while this command is easy to use, having an HTTP interface is more convenient. Let’s recycle the VCL code we used earlier for purges and implement some ban logic:
vcl 4.1;
sub vcl_recv {
if (req.method == "PURGE" && req.http.x-invalidate-pattern) {
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host);
return (synth(200,"Ban added"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
Performing a ban
The HTTP interface is similar but requires passing an “x-invalidate-pattern” header that contains the URL pattern we’re matching. The “host” header is used to match the host.
Performing the ban can be done using the following HTTP request:
PURGE / HTTP/1.1
X-Invalidate-Pattern: ^/admin
Host: example.com
The corresponding cURL command would be as follows:
curl -XPURGE -H"X-Invalidate-Pattern: ^/admin" http://example.com/
Besides the “-X” option we covered earlier, there’s now a “-H” option to include an extra HTTP header. This header is our “X-Invalidate-Pattern” header that contains the “^/admin” pattern.
But again, you’ll probably use the HTTP client of your development framework or a CMS plugin that handles bans for you.
The ban lurker and asynchronous bans
Bans are processed by the “ban lurker”, which is a separate thread that manages the ban list. While bans can be executed in a request context, the ban lurker can also process ban expressions outside of the request context.
This means the ban lurker has no request information available and can only match the properties of the cached object. For example, the “req.url” or “req.http.Host” variables are not available to the ban lurker, but for example “obj.status” and “obj.http.Content-Type” are available.
If you want to ban content based on the URL, you’ll need to perform some VCL tricks to have so-called “asynchronous bans”.
Let’s look back at the initial VCL example for bans and dissect the elements that are used to enforce asynchronous bans:
vcl 4.1;
sub vcl_recv {
if (req.method == "PURGE" && req.http.x-invalidate-pattern) {
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host);
return (synth(200,"Ban added"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
It starts in “vcl_backend_response” where we are injecting request information into the object. The URL and Host header are typically not available in the response context, that’s why we’re creating the custom “x-url” and “x-host” response headers.
In “vcl_recv” we’re then using the “obj.http.x-url” and “obj.http.x-host” variables to match the request URL pattern and Host header. The complete ban expression would look like this:
obj.http.x-url ~ ^/admin && obj.http.x-host == example.com
The ban lurker can match these values and remove the relevant objects from the cache. The ban lurker also has some runtime parameters that influence its behavior:
- “ban_lurker_age” has a default value of 60 seconds and defines the minimum age of an object before it can be considered for banning
- “ban_lurker_sleep” has a default value of 10 milliseconds and defines how long the ban lurker sleeps before it continues inspecting the ban list again
- “ban_lurker_batch” has a default value of 1000 and defines the maximum number of ban expressions that are processed in a single batch
How the ban list works
The ban list can be viewed by calling the “varnishadm ban.list” subcommand. This is what it looks like initially:
varnishadm ban.list
Present bans:
1603270370.244746 0 C
Even if no bans have been added to the list, there’s always an initial ban as a point of reference. A ban is identified by its UNIX timestamp and this one has zero objects that it matches. The “C” symbol indicates that the ban has been completed.
Let’s add a new ban to the list:
varnishadm ban obj.status != 0
When we run “varnishadm ban.list”, you can see the newly added ban:
varnishadm ban.list
Present bans:
1603272627.622051 0 - obj.status != 0
1603270370.244746 3 C
Meanwhile, 3 new objects have been added that have seen the initial ban, but still need to work their way through the list to match newly added bans. As time passes, the ban eventually completes and the ban list looks like this:
varnishadm ban.list
Present bans:
1603272627.622051 0 C
1603270370.244746 0 C
Here’s another example:
varnishadm ban.list
Present bans:
1603273224.960953 2 - req.url ~ ^/[a-z]$
1603273216.857785 0 - req.url ~ ^/[a-z]+/[0-9]+
1603272627.622051 9 C
This example features 2 bans that were added to the list and the initial one that is used as a reference.
The initial ban has 9 objects that have to work their way through the list. Zero objects have been processed by the first new ban, but 2 objects have reached the top ban.
Securing your bans
Just like purges, bans also need to be protected. We’ll be using the same access control list to do this. Here’s the initial ban VCL example, with more security:
vcl 4.1;
acl purge {
"localhost";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE" && req.http.x-invalidate-pattern) {
if (!client.ip ~ purge) {
return(synth(405));
}
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host);
return (synth(200,"Ban added"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
Ban limitations
Although bans can remove multiple objects in a single call, unlike purges, they don’t scale well. A ban needs to examine every object to see if there’s a match. The more objects in the cache, the more CPU-intensive this operation is.
If we factor in large ban lists, the number of operations equals the number of bans times the number of objects.
At scale, an alternative solution is required to remove multiple objects from the cache at once.
Secondary keys
Secondary keys can be that alternative. As the name suggests, secondary keys offer an additional way to identify and remove objects from the cache. Custom keys can be used to tag content and invalidation can be done based on these keys.
Varnish has 2 modules to do this:
- “vmod_xkey”, which is on open-source VMOD
- “vmod_ykey”, which is an enterprise VMOD
Let’s have a look at their capabilities.
The xkey VMOD
Vmod_xkey is an open-source module that you can download from https://github.com/varnish/varnish-modules. Currently, there are no packages available, so you’ll have to compile the module from the source.
Before we dive into the VCL code, I’m going to show you how to register keys using the “xkey” HTTP response header. Here’s an example HTTP response:
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Cache-Control: public, max-age=600
Xkey: category_sports id_1265778 type_article
Besides the obvious “Cache-Control” header that sets the Time To Live to 10 minutes, there’s also the “xkey” header. Although it contains multiple values, it’s presented as a single string with space separation.
This response registers the following keys:
- “category_sports” which identifies the category of the page
- “id_1265778” which is the unique identifier of the page
- “type_article” which tags the page as being of the type “article”
Once the keys are registered, the “xkey.purge()” function can be used to remove objects based on these keys. Here’s an example:
vcl 4.1;
import xkey;
import std;
acl purge {
"localhost";
"192.168.1.5";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405));
}
if(!req.http.x-xkey-purge){
return (purge);
}
set req.http.x-purges = xkey.purge(req.http.x-xkey-purge);
if (std.integer(req.http.x-purges,0) != 0) {
return(synth(200, req.http.x-purges + " objects purged"));
} else {
return(synth(404, "Key not found"));
}
}
}
Just like the other examples, this one also uses the “PURGE” request method and is compatible with regular purges. When the “x-xkey-purge” header is not set, it can fall back on regular purges using the URL and host header.
If “x-xkey-purge” is set, the keys it contains are passed to “xkey.purge()”. The output of that function is stored in the local “x-purges” header which is represented by “req.http.x-purges”. If the value is zero, the key wasn’t found and a synthetic 404 response is returned. If the value is not zero, the xkey purge has matched a number of objects and removed them from the cache.
Soft purges are also supported by using the “xkey.softpurge()” function instead of “xkey.purge()”. The existing “grace” and “keep” values are used for xkey softpurges.
xkey limitations
The xkey VMOD is quite limited in its feature set. You can only use the “xkey” header or alternatively the “x-hashtwo” header to register new keys. Multiple keys can only be separated by spaces or commas.
Xkey doesn’t offer the ability to register explicit keys in VCL without the use of headers. And while it performs better than bans, it’s not optimized to work with really big data sets.
And that’s why we developed the ykey VMOD.
The ykey VMOD
The ykey VMOD is Varnish Software’s response to the limitations of xkey. It’s a commercial VMOD that is part of Varnish Enterprise.
It was specifically designed to handle massive data sets and be compatible with our Massive Storage Engine, which can hold terrabytes of data.
Additionally extra features were designed to allow more flexibility in the way tags are registered and objects are invalidated.
The first VCL example I’m going to show you is not related to invalidation, but to the flexibility of registering tags:
vcl 4.1;
import ykey;
sub vcl_backend_response{
ykey.add_header(beresp.http.Ykey);
ykey.add_header(beresp.http.Xkey);
if (beresp.http.Content-Type ~ "^image/") {
ykey.add_key("image");
}
}
This example allows you to set the header that will be inspected for tags during object insertion. Through the “ykey.add_header()” function we’ve opted to use the “Ykey” header.
For compatibility reasons we also added the “Xkey” header. Instead of solely relying on headers for key tagging, we explicitly register the “image” tag for objects whose content type matches the “^image/” regular expression. This means we are looking at image MIME types such as “image/jpeg” or “image/png”.
Let’s throw in an invalidation example:
vcl 4.1;
import ykey;
import std;
acl purge {
"localhost";
"192.168.1.5";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405));
}
if(!req.http.ykey-purge){
return (purge);
}
set req.http.x-purges = ykey.purge_header(req.http.ykey-Purge, sep=" ", true);
if (std.integer(req.http.x-purges,0) != 0) {
return(synth(200, req.http.x-purges + " objects purged"));
} else {
return(synth(404, "Key not found"));
}
}
}
In this code snippet, we’re not beating around the bush: the ACL is immediately included and enforced. There is still backward compatibility for regular purges if the “ykey-purge” header is not set.
If “ykey-purge” is set, we’re using the header in the “ykey.purge_header” function that takes the header name as the first function argument, the separator as the second argument, and a soft purge toggle as the third argument.
In this example, we’re invalidating the objects whose tags match the value of the “ykey-purge” header. If multiple tags are passed, we’re using space as the separator. And finally, we’ve chosen to use soft purges. By omitting the third argument or by setting it to “false”, regular purges are used.
Reach out to us for more answers
If you have any further questions don’t hesitate to reach out to our team of experts.
