How to use caching to fence off hurricane traffic

It’s hurricane season, and the Caribbean and southeastern United States are experiencing some of the most devastating weather events they’ve ever seen.

It might not seem like the time to be worrying about stuff like web traffic, but guess what? Major weather events actually drive unexpected masses to certain websites. In a weather situation or disaster, a whole lot of people are probably trying to access sites that will give them the latest news… and the recent Hurricane Harvey in Texas was no exception.

Varnish is no stranger to this kind of, if you’ll excuse the pun, flood. You can read, for example, our case study about how Surfline, a website that started to share weather and surf conditions but ended up being the go-to, real-time coastal weather condition site, uses Varnish for maximum uptime no matter the level of traffic. In big high-traffic events, Varnish is a lifesaver because of its advanced caching technology, high availability and the ability to manage whatever traffic (or weather) is thrown at it.

With our background in this unique kind of disaster management, we were interested in Lee Hutchinson’s article at Ars Technica, “How to hurricane-proof a web server”, and thought that anyone interested in resilience, uptime and web performance under any and all conditions would be, too. Hutchinson writes about Space City Weather, a Houston-based weather website/blog that gets maybe 5,000 visitors and 10,000 page views per day - maximum. But with Hurricane Harvey on the horizon, that traffic ballooned to over 1.1 million views on August 27, and each day between August 24 and 29, there were at least 300,000 views. Exponentially more than normal, and certainly not expected. That’s where the technology comes in, and in this case, quite basic technology that is often overlooked: caching. Hutchinson’s article, in fact, states:

“I’d like to be an immodest rock star sysadmin and claim that I customized my hosting specifically around some prescient guess at SCW's worst-case traffic nightmare scenario, but that would be a lie. A lot of what kept SCW up was a basic philosophy of ‘cache the crap out of everything possible in case traffic arrives unexpectedly’.”

It so happens that, according to the article, Varnish was the secret weapon, handling most of the traffic that made it to the Space City Weather server. It was instrumental in ensuring that the site could scale to more than 100x its normal daily load.

Ready to learn more about Varnish and caching? Read the O'Reilly Getting Started with Varnish e-book.

Photo by Witch Kiki on Unsplash