Keep your stack up-to-date (we can help!)

Let’s open with an extremely controversial statement: you should run up-to-date software*. I think everybody here agrees with that, the same way people agree with “up-to-code buildings are better” or “eating a party-size bag of Cheetos in one sitting isn’t healthy”. However, because we should do something, even if we know it’s objectively good for us, doesn’t mean we will.

But, if we understand what is at stake, how we got there, and how to fix it, I have the highest hopes that things will be a lot better. This post will be centered around Varnish, of course, but most of the points we’ll discuss here will apply to every piece of software.

Security is unimportant, until it hits you

You know how machines just inflexibly run the code that fallible humans write? And you know how sometimes all hell breaks loose because “somebody did something somewhere” and your system wasn’t ready for it?

Well, for all our faults, we humans learn from our mistakes, and when something goes wrong, we tend to fix it. Even if things aren’t bad but we think they could be better, we do something about it. That’s what software versions and patches are about: a new version means fixed bugs and new features, usually a new version is better (inb4: Windows, don’t @ me).

The features are not my concern here, because if you don't need a specific functionality, or you can “live without it”, then you don’t need to upgrade because of it.

Bug fixes however, we should care about, deeply. Especially because backdoor injection is a prominent threat now. In the Varnish community, backdoors are not really a worry because how tightly-knit our community is, and the Varnish project is really carefully about what dependencies it pulls in (<cough>npm</cough>)

Bugs are still a software reality, though,  and CVEs are something to watch out for. If you are new to the space, CVE are basically bugs that can break a service or be an attack vector. Thankfully, the software community has ways of finding and fixing them before they are weaponized by bad guys™, even if 0-day vulnerabilities are a danger.

Why am I saying all this? If Open Office has a bug that prevents you from opening a spreadsheet, that’s a problem, but if your OpenSSL implementation has a bug, that can be a BIGLY YUGE problem, and if you don’t update while a fix is available, you’re opening yourself (and your service, and your potential company) to a world of pain.

Note: sorry if I’m being a bit of a fear-monger here, but it’s really important to convey this point across: bug fixes are useless and can’t save you if you don’t deploy them.

Oh, and there’s legal liability too!

I’ll admit I learned about this one recently (which prompted the writing of this post), but there are legal implications to not deploying fixes. In the US for example, there’s Executive Order 14028 which makes diligent security updates a conformance thing.

It’s a bit akin to your HOA telling you that your big tree that’s leaning over the neighbors yard is a safety risk, and you should really do something about it. Even if you could technically suffer the consequences of a breakage, there’s no escaping the law.

Just upgrading may not be enough??

For this one, we need to take a tiny step back and explain a bit how software is distributed. Feel free to skip it if you already know all about it.

Usually, software is published as source code (in the open-source industry at least), but it doesn’t magically appear on the user's computer, ready to run. In the pipeline, there’s a packaging process, sometimes multiple, for example, if you are using Ubuntu or Redhat, there are multiple teams and individuals in charge of grabbing the source code, compiling it and pushing it to repositories that will serve it to users.

That’s where we need to look: Linux distributions will pick which versions of the software they will package, and all have various policies regarding what justifies a package update. Rolling distributions (BTW I use arch) will often pick the absolute latest code published by a developer, and Debian (and therefore Ubuntu) or Centos will only consider bugfix versions once the distribution is released.

This means some “tracks” will keep old versions around for a long time (we discussed this previously, in the Noble packages post), which may be an issue because they are “blind” to some very important updates.

To the problem, there’s a fix

Varnish, like others, have a predictable but fast-paced release schedule: twice a year, plus emergency bug fix releases in case of a CVE. However, the bug fixes only apply to “current” releases. In our case, that means releases that are younger than a year old (and our LTS version, we’ll get back to that). For Linux distributions that have a longer life cycle than 12 months, it becomes an issue.

The way we personally solved this problem is by offering another track, using community packages. With this, you can get current packages, even if your distribution of choice doesn’t support them. And they are easy to get, too:

Simple, right?

With this, you are able to stay on top of updates with minimal effort. Also, note that in Varnish’s case, we are careful to minimize API changes, so switching from one version to another generally means no configuration changes, just install the new binary, restart and go.

The joy of LTS

If you feel that switching to a new Varnish version every 6 or 12 months is still too frequent a task, we got you covered. Varnish Software maintains a Long Term Support (LTS) version. Essentially, it lives on for years and while you don’t get new features, you still benefit from the applicable bug fixes that get backported from newer releases. And you install it the same way as other releases:

I call it the “boring” version, because it just works, there’s no new functionality, but it's rock-solid, and apart from upgrading regularly, you have nothing to do. Very boring.

The LTS is also the basis for Varnish Enterprise, but we ALSO add new features, so you get the best of both worlds: cool new features, all the API/configuration stability and the bug fixes (that’s technically 3 worlds, but let’s not complain about the abundance of good stuff).

Support and upgrade path

Of course, if you are using a 10 year old Varnish release, some things will have changed (for the better, mind you). The VCL syntax got tweaked, some parameters were renamed, and there are now probably better and faster ways to solve your problems of yore.

Should you be in that situation, be aware that we offer support, both for the open-source Varnish Cache, and for Varnish Enterprise. We can help you upgrade, clean up and optimize Varnish specifically, and your service in general, be it a website, API service, or any HTTP-based platform in general (we do a bit more than caching, after all).

Anyway, I thank you for taking the time to read, and I hope I managed to convey how crucial it is to protect yourself through regular updates, not just for central pieces of software like Varnish, but throughout your infrastructure. Not only that, but on-top of pointing out potential problems, I was hopefully also able to offer clear solutions on how to alleviate the pains of fixing them.

If you'd like to know more about all our support offers (open-source and Enterprise), you can check out this page or contact our team directly. And to get your toes wet, you can also find us on Stack Overflow, Discord, mailing-list and more!

* Did I write “controversial”? I meant “obvious”.