Greetings from Denmark: Edge-side HTTP Security Toolkit

Last Wednesday I was at a breakfast seminar in Copenhagen hosted by Casalogic and Varnish Software at the very cozy Admiral Gjeddes Gaard in the center of the city.

We had many attendees and the interest in our software and services was huge. Kristian Lyngstøl of Core Dev fame held a great talk about Varnish Cache and had great answers to all questions from the crowd. From Casalogic, Daniel Schledermann spoke a bit about Varnish Cache in a web architecture context and explained how it has made things easier for customers such as Novasol (Northern Europe's largest company for holiday rentals with 28,000 holiday rentals in 23 European countries). Lastly, a talk about our company, commercial services and especially the Varnish Administration Console was held by yours truly. Under that talk I touched on a subject you do not hear so much about: How Varnish Cache, as a tool, can be used to protect your web infrastructure from different types of attacks.

So, here is my second take on the blog series highlighting Varnish Cache and its usage in the real world, solving real problems.

As stated in my last post, HTTP is the universal transport protocol and the only universally deployed API accross devices, clients, servers running all kind of operating systems. As part of my work at Varnish Software, I am in daily contact with users doing lots of ingenious things with Varnish, beyond caching. One of the coolest things you can do, when thinking outside of the box, is using Varnish Cache as a means for website security.

IT security, being one of my favorite subjects, is a neverending novel/story. This year you have for sure read news stories about websites getting p0wned by all kinds of stakeholders ranging from highly skilled black hats and idealists of all sorts to script kiddies and crackers using relatively simple techniques, such as SQL injections and DDOS attacks that you can find HOW-TOs for on the interwebs. Honestly, I envision a world where I could promise you that by using our software that will not happen to you, but unfortunately that world does not exist! What I can do is give you some tips on how you can reach a higher level of security protection for your website by using Varnish Cache and mitigate some of the worst effects such attacks carry with them.

So I will give you a walk-through of the secure-by-design concept and how it applies to Varnish Cache, put forth the theory of Fast == Secure, hint: fending DDoS attacks, and finally set the spotlight on two projects living out there: Security.VCL and Varnish Application Firewall. Hopefully, you will learn something new by reading this and have lots of fun while using Varnish Cache beyond simple caching, making it your next-generation Web Application Firewall and DDOS attack defender!

Security is never a coincidence

• The really important barrier: The central actor in Varnish is the manager process, "MGR", which is the process the administrator "(ADMIN)" starts to get web-cache service.All management happens in one process, all actual movement of traffic happens in another, and the manager process does not trust the child process at all.
• The Admin/Oper BarrierVarnish operates with separate administrator "(ADMIN)" and operator "(OPER)" roles. These days operators are often scripts and data collection tools, and there is no reason to assume they are bug-free, so Varnish does not trust the operator role, that is a pure one-way relationship.The administrator has the final say, and of course, the administrator can decide under which circumstances that authority will be shared.
• Other barriers: There are more barriers that are more sort of "technical" than "political" and generally try to guard against programming flaws as much as security compromise. You can spot them by following the arrows in the diagram found in the documentation.

Fast == Secure

This one is by no means obvious, but after thinking about it twice it makes sense. Per Buer, Varnish Software CEO, has written about this in our Varnish security whitepaper so here too I am going to simply give you a quick overview:

Using Varnish to fend off DDOS attacks

Handling the volume

Some attacks use real browser sessions to send requests. These are impossible to distinguish from real user traffic. If the attack is small or mid-sized you might actually manage to handle the attack by adjusting your caching policies and making sure your hit rate is approaching 100 percent.

Having specially crafted VCL ready, which sets the TTL of all objects to a week, might mean that you’ll serve stale content - but at least your site will stay up. Take special care to make sure various forms and other POST requests get handled in an appropriate manner.

But please, do not believe me, let real users tell you:

Entropy DDoS detection

Now, this is my take on the academical type of thought experiment:

You have an engine to handle request lightning fast, Varnish Cache, and then you have a way to somehow automatize the DDoS detection and trigger a change in the rules or at least give someone somewhere an alert by analyzing and calculate the simple entropy. Something like:

Entropy  H(P1) + H(P2) + ... + H(Pi) > H(P1P2..Pi) => DDOS

That would be a cool PhD dissertation, right? Well go crazy and read more.

Web application firewall, you said?

Web application firewall (WAF) was a term coined a few years ago describing a firewall that is operating on the application layer. These firewalls act as filters looking for suspicious patterns and blocking them out.

The most famous and widely used open source implementation of this concept is by far the Apache Mod_Security project. It has everything to be a great tool, but lacks one thing that Varnish Cache has: Speed. As far as I know there are two projects working on different approaches into making Varnish Cache a useful WAF, and they both take Mod-Security as their starting point: Security.VCL and Varnish Firewall:

Security.VCL - An Apache mod_security replacement?

The aim of the Security VCL project is not to replace, but rather to complement Apache mod_security by adding a layer of security in your edge server and throws poisonous request to the trash bin, before they hit your backend servers and eat your resources.

In 2008, we took the predefined patterns implemented in by the insightful people of mod_security and rewrote them to VCL. Since VCL is compiled to native code, we are able to run most of these filters with almost no performance impact. These patterns look for attack signatures in URLs or headers. A lot of commonly used SQL injection attacks and access to deployed root kits can be stopped this way, either rendering the attacks futile or preventing the black hats from actually taking advantage of your servers.

As Varnish is deployed at the edges of your network, implementing changes in your Varnish configuration is much less error-prone than deploying changes to your core web applications. In most cases, a change in Varnish can often be deployed much quicker than a full blown deployment of your web application. Often the change is minimal, blocking a certain set of URLs or adding missing escape characters. Then, after you have deployed the hot fix in Varnish you can work on doing the proper fix in your web application without shutting your site down for hours.

Varnish Firewall - Replacing Apache mod_security

This time the answer is, well, yes. And, talking about academic thought experiments, this is the subject for a master's thesis.

This project has set the mark high, very high. The Varnish Firewall aka OWASP VFW, was started in 2011 by Eduardo S. Scarpellini, Brazilian master's student, as his thesis. The purpose of the project is to use "Varnish as a Web Application Firewall; A kind of mod_security for Varnish; Varnish security filters."

As the Open Web Application Security Project aims to "Be the thriving global community that drives visibility and evolution in the safety and security of the world’s software." I believe VFW lives in the right place on the internet and as such this project is a very interesting showcase of what Varnish Cache can currently do and which direction future development of the software can take.

Image (c) 2008 Burkhard Schmidt, published using Creative Commons (CC) 2.0 license.