In Dridi's post, you learned about Varnish's technical choices regarding backends, and why a VMOD was necessary. If you didn't read the article (shame on you), here's the recap:

• Varnish resolves domain names and IP when compiling the VCL, so the resolved address is set in stone.
• And there must be exactly one resolved address, if the domain name is not found, or is served by multiple IPs, the VCL won't load.
• You can work around this with a cron'd script, periodically generating a VCL with the correct and updated addresses.
• However, this forces you to garbage-collect the old VCL; you'll have to take extra care of the probes and you'll reset the backend stats everytime you reload since backends are tied to the VCL.
• So we built a VMOD.

And that VMOD is named "named", another proof that engineers shouldn't be allowed to name things.

The vmod that should not be "named"

vmod-named is made to be a drop-in replacement for the DNS director we had in Varnish 3, thus helping  our users to transition smoothly to version 4 and upwards.

Note: Varnish 3 has been EOL on the community version for a long time now, and we will unplug life support from Varnish Cache Plus 3 pretty soon, so please, migrate! We can help you with that, life is too short to be stuck on v3.

The thing is, we didn't stop here, because vmod-named, as nice as it is, has a specific mission: provide a familiar interface to the DNS director, and so is focused on DNS. But the cloud/container crowd cried "More! More! More!" (with a rebel yell, I'm guessing), and so, we created a second VMOD: vmod-goto.

Evanescent backends: Bring them to life

The API is a tad bit more direct than named's, mainly because you don't need to create a director:

sub vcl_backend_fetch {
		set bereq.backend = goto.backend("www.example.com", 80);
}

This creates a pool of backends that may exist only for a few seconds. After each TTL period (10 seconds by default), goto will update the pool, or destroy it if no one used it during the last period. This ensures that unused backends are garbage-collected and that we keep the memory footprint as small as possible.

Note: Revalidation is done asynchronously so updating the backends is done in a non-blocking manner.

The first VCL snippet gave you an idea of what you can do, but let's go further:

sub vcl_backend_fetch {
    	set bereq.backend = goto.backend(bereq.http.host);
}

This basically amounts to transforming Varnish into an HTTP reverse-reverse-proxy (or maybe "forward-proxy", if I dare to coin a term here). goto will resolve the domain name for you, recognize the port and even create an HTTPS if the host header started with "https://", and make it super easy to use.

This is very nice because you can connect to a backend you know nothing about beforehand, meaning for example that you can let your APIs dictate the backends to use. This will notably be used in the next iteration of Varnish High Availability and of the API Engine to provide super clean VCLs and avoid reloads.

I hid quite some parameters in my goto.backend() calls, but for good reason: all but one are optional. Leveraging the VMOD facilities, we can provide sensible defaults, allowing you to keep a short and clear VCL in the general case, while allowing you to fully express what you need in the specific cases. You can for example specify the port, or the TTL of the domain names, or as in named, the probe to use to make sure the announced IPs are up.

Evanescent backends are like fingerless hands: you can't count on them

With Varnish 4, we gained one awesome feature: director stackability. I talked about it in a previous post, but as a refresher, Varnish allows you to put a round-robin and a last-resort server inside a fallback director. The idea is that, as long as the round-robin has a healthy backend, it will get picked, and if the round-robin can't deliver (pun intended), the last-resort server is used.

This offers a level of versatility that is hard to beat, based on a very simple logic, which increases maintainability, perfect!

But, how can we do this with goto? The backends returned can be destroyed at any time if not used, so it seems like a very bad idea to put them in a director and to rely on them. And that is indeed a terrible idea, which is why we have another function: goto.director().

It accepts the same arguments as goto.backend(), and will also return a pool of backends, BUT it also promises that the pool will never be garbage collected (it may be empty though), making it safe to call it like this:

backend lastresort {
        .host = "192.168.1.200";
}
    
sub vcl_init {
    new fb = directors.fallback();
    
    fb.add_backend(goto.director("https://api.internal:8080"));
    fb.add_backend(lastresort);
}

To boldly goto where no vmod has gone before

goto is already a very useful VMOD, but we are not done yet. It started a bit as a named copycat, however the intended scope is way broader because goto is not about DNS, it's about discovering dynamic backends easily. And to achieve this goal, we are hard at work bringing SRV records and Consul.io support for the next releases, so stay tuned.

And since Dridi quoted Star Trek in his post, I feel  obligated to fly my trekkie flag high and salute you the Vulcan way: peace and long life.

You can learn more about our latest Varnish Plus release from our CTO, Per Buer, during a live, January 17th webinar. 

Photo (c) 2014 Erik Wilde used under Creative Commons license.