Celebrating 10 years of Varnish Cache: Q&A with the man behind the idea

In 2005, VG Nett in Norway reached over 45 million page views every week, half of these on the front page. Twelve servers were needed to handle the requests. Anders Berg, a system administrator at VG, believed that there must be a way to reduce the number of servers needed and to load the pages faster. He wrote a spec on which Varnish Cache, the HTTP web accelerator, is based. 

This is the first in a series of blogs celebrating the open source community leaders who have made the biggest impact upon Varnish. In this Q&A Anders Berg, now a business owner, shares with us the short version of how the idea of Varnish Cache came to him. 

Anders starts the conversation by explaining that he is a Myers-Briggs ENTP personality type. “I am a thinker, not a doer,“ he explains. “I came up with the idea of Varnish Cache but there were others who executed it.”

What were the challenges you were dealing with at the time?

The online news site vg.no, was launched in 1995. It was made by hand in HTML, served by Apache and used Squid for caching from very early on. As the news site grew, more and more Squid servers were needed to keep the site from crashing from the ever increasing visitor traffic. The sys admin team, including myself, was very busy with simply maintaining Squid and keeping the system, as well as the site, up and running. We were always worried that the site might break, and,in fact, it often did, that’s how fragile the system was. I kept thinking that there must be a more effective way of caching and accelerating the site. So, I contacted the manager of the Squid project to figure out if there was a way to develop Squid to make it perform better. But we concluded that the quality of the Squid code base was not good enough to develop into a caching product that could meet our needs.

How did the idea of Varnish Cache come about?

Vg.no primarily used open source software to run their site at the time. I wanted to give back to the open source community, and I figured a good way to do that would be to develop and introduce a great open source caching solution. However, to accomplish something like that we needed someone dedicated to open source and preferably someone recognized within the open source community to join us on this journey. A fellow sys-admin at another Norwegian newspaper, Anders Nordby, recommended I reach out to Poul Henning Kamp*, who was responsible for the first generation crowd sourcing of the freeBSD project and an engaged freeBSD man. Anyone working in the open source world knew that there was a gap in open source software when it came to front-end delivery (as Squid was the only existing solution at the time). Anders adviced that we should develop a solution that was as fast as possible. He suggested that we should get a kernel developer for this task, someone with deep kernel knowledge who could utilize modern features of Linux/FreeBSD kernels properly. Anders knew Poul Henning and suggested he might be interested in this project. I wrote him an email that included the premises for the project to ask if he was interested in being a part of it. The important elements were among other things that this had to be an easily configurable http accelerator. Poul Henning was interested in  the idea, and as soon as he was a part of the picture I introduced the idea of creating our own open source software to vg.no’s management team.  I made sure they were aware of the fact that because of open source we were running our site at a very low cost and managed to get them to commit a budget to the project. We submitted the first piece of code to the Varnish Cache project repository on February 22nd 2006, and the rest is history.

When did you start using Varnish Cache?

We tested the project live on the vg.no news site, and by September 2006 we were running on Varnish Cache alone. We’d removed all Squid servers from our infrastructure. Our site ran smoothly handling massive spikes in traffic on three servers instead of 12.

Why did you decide to develop Varnish as a reverse proxy?

The idea was to cache from the outside in, rather than the inside out as Squid does.

How did you decide on the name Varnish Cache?

I thought of the name when I was on sick leave because of a prolapsed disc in my back. During that period I spent a lot of time in a room where we had a poster on the wall that said Vernissage, which in french means the preview of an art exhibition. I was intrigued by this word and looked it up in Merriam-Webster's where it’s said to mean deceptively nice surface. I figured that was the right description of our software since Varnish makes even the worst CMS deliver making it perform deceptively nice as well.

What do you think of Varnish Cache as it is today?

It makes me extremely proud to follow the activity around Varnish. Website traffic has taken off completely, and I feel the software has been developed in the right direction to meet the needs of the market and the future.