The backend of backends: Scaling for the junk in the trunk

When PAPER magazine published a series of nude Kim Kardashian photos in winter 2014, throngs clamored to see photo spreads of her, erm, assets. Most of these eager viewers, like all internet users, wanted to click and see immediately. They don’t give a second thought to what enables the backend to handle the traffic coming to see the... back end.

Most people reading now and who take interest in website scalability and content availability do think about that part of the equation. In fact for online media outlets, some event or content will contain breaking news or something of such great public interest that even if it doesn’t, as Kim’s PAPER spread threatened to do, “break the internet”, it could put significant strain on servers and ultimately take a website down. The Message, an online writing collective that is a part of digital publishing site Medium, dedicated an entire article to explaining the web architecture considerations that went into keeping Kim live and accessible (and, as the article highlighted, and where Varnish Cache comes into view: “Preparing for Kim; or: cache rules everything around me”). Luckily, everyone in the chain knew they were looking at unprecedented traffic numbers and would need to get ready. They predicted at least 100 million page views, and at each step asked themselves: would their existing infrastructure, which easily supported the couple million people per month who visited the PAPER site under normal circumstances, perform under extreme conditions?

That is the kind of question any large media outlet is wise to ask. When a publication or website secures an exclusive and highly in-demand piece of content - that’s only half the battle. The technical side shifts into gear at that point, as it did for PAPER: what good is the most talked about piece of content in the western world if it crashes the site and no one can see it?

As the Message article recounts in great detail, the backend is complex and scaling comes down to a tailored software stack that meets the needs of the media outlet, its content and its users. While all of this boils down to the simple point that infrastructure setup should be customized to the needs of the individual media outlet, it’s important to see that whatever setup, it must be invisible to the end user, whose experience is paramount.

Learn more about Varnish Software solutions and website scaling and availability so you never get caught with your trousers down.

Image is (c) 2014 Paul Costanich.