Celebrating 10 years of Varnish Cache: Q&A with the Chief Architect

In this first part of two posts, Poul-Henning Kamp looks back on the genesis of Varnish Cache and what makes it what it is today. (Read part two of the PHK discussion here.)

Ten years ago you were asked to contribute to Varnish Cache. How did that come about?

Dag-Erling Smørgrav pulled me aside at a EuroBSD conference in Basel and suggested it, that must have been late 2005. It certainly does feel like 10 years: My kids have grown up and moved away from home in the meantime. So, yeah, it’s been a long time.

What made you interested in Varnish Cache?

To be honest, I wasn’t very interested. I was a core FreeBSD kernel programmer, not somebody writing web servers.

But for a number of years I’d been annoyed by the fact that we’d been doing all this work in the Unix kernels to make them work better on modern hardware, and people were not using any of it. They were still acting as though these were computers from the 1960s or 70s.

So I thought, what the heck, I can write this web proxy and then I can point at it and say that’s how you are going to get performance out of your modern operating system.

However, I don’t think many of the clues in Varnish have been picked up elsewhere. For instance, I haven’t seen the shared memory logging anywhere else yet.

What has been the most difficult stage in developing Varnish Cache?

That was the release cycle from Varnish 3 to 4 where the project grew up. I should probably have recognized that because we had the exact same thing happen in FreeBSD between versions 4 and 5.

Varnish started out with devops people, who were running the website and programming. If we asked them, “Can we just test this new feature?” somebody would say, “Sure, I'll throw it in production and see what happens.”

But then, as projects grow, people get dragged out of operations and become full-time developers, and suddenly you have lost the ability to do informal live-tests.

None of our core developers run a Varnish server of any significance any more, and that changed the dynamics of testing a lot and delayed the Varnish Cache 4.0 release: We had to persuade people running Varnish servers to test the new code in production.

What stands out over the past 10 years in this project?

It’s humbling to see how many people rely on Varnish.

I’m just this dude from a small one-duck-in-the-pond village, the Danish equivalent of Hoople, North Dakota, and 13% of the top 10,000 websites in the world are running my software? I’ll never quite get used to that.

I’m a big fan of the performer Amanda Palmer, one half of the Dresden Dolls. She once gave a very good commencement speech about this nagging feeling when you go to bed at night, that at 3 am, someone will knock on your door saying, “This is the fraud police: You’re just making shit up as you go!” I can totally relate to that.

In these 10 years what has made you most proud?

Inside the Varnish project it’s probably the VCL language. I shared the original Varnish design documents on the day of the 10-year anniversary, and interestingly, the first design doc has many other things, but it does not have the VCL language.

In the early days of Varnish, as we talked back and forth, Anders Berg from VG Multimedia would say, “Oh, I also need to be able to control this, and I need to be able to look at that, and do this.” I had the idea that it would be simpler to write him a programming language. That way he could write his weird policy code himself.

That was the lightning strike: If I just wrote this little compiler, people could do whatever they want with Varnish, without my help or involvement.

Overall, I think that’s the biggest success in Varnish Cache: You can write up what you want to happen, we compile it to machine code, and you can run it at a million requests per second, and it still works. That was a good idea.

Read more in part two of our Q&A with Poul Henning.