Return to Blog
April 17, 2013
5 min read time

Withstanding DDOS attacks with Varnish and COTS hardware

Per Buer

Recent years denial of service attacks have become commonplace. You no longer have to create your own botnet in order to attack someone. A PayPal account, some Googling skills, a couple of forum posts you can have a 400,000 PPS SYN Flood going in an hour or so.

Withstanding such an attack is problematic. First of all, you’ll need the bandwidth and networking equipment to actually handle the massive amount of packets flooding your pipes. If the network holds the next thing to break would be the load balancer or caching layer. This is where Varnish lives and over the years some of you have asked for help dealing with various attacks.
In order to sum up some of our experience I thought we’d publish some of it here. If you have anything to add please use the comment section below.
I’ll go through the following types of DDoS here:

  • SYN Floods
  • HTTP Floods

Only the latter is really Varnish related but there isn’t too much information on how to deal with SYN Floods at any scale so we’ll share what we’ve got. We still see quite a bit of UDP based floods from time to time, but there isn't much one can do once the traffic has entered your network.

HTTP floods

These are request floods. They try to overpower your caching or backend layer with sheer brute force. Since HTTP requests are rather small and the response is rather big a rather small stream of requests might fill up a rather big pipe with responses. So, even if your server is capable of serving the request it might not have the network capacity to do so.

The first step towards mitigation is detection. You need to identify which requests are part of the attack. varnishtop -c -i rxheader will give you a live updated list of the most popular client request headers. Look for anything that stands out. Have a look at output from varnish-cache.org:

 3417.25 RxHeader       Host: repo.varnish-cache.org

 2785.50 RxHeader       Connection: keep-alive

 2476.50 RxHeader       Cache-Control: max-age=0   

 1907.50 RxHeader       Host: www.varnish-cache.org

 1818.75 RxHeader       Connection: close

 1793.50 RxHeader       X-Forwarded-Proto: https

 1398.00 RxHeader       Accept-Encoding: gzip, deflate

 1236.25 RxHeader       Accept: */*

  944.75 RxHeader       Referer: https://www.varnish-cache.org/

  753.50 RxHeader       User-Agent: Debian APT-HTTP/1.3 (0.8.10.3)

  544.75 RxHeader       Accept-Encoding: identity

  503.75 RxHeader       User-agent: urlgrabber/3.1.0 yum/3.2.22

  444.75 RxHeader       Accept-Language: en-US

This gives you a good overview over the most popular incoming headers. Obviously a big part of the traffic is requested by Debian APT-HTTP and the Yum urlgrabber. Now, since varnish-cache.org has a couple of software repositories on it this is normal but the idea here is to look for anything that stands out. User-Agent, Accept-Encoding, Accept-Language are typical tell tale headers. Once you think you’ve spotted something fishy switch back to the full varnishlog output to verify that the requests are actually bad.

What to do with malicious clients?

There are several things we could do with them. The simplest thing would be to throw an error:

This would free Varnish from delivering the full object and any assets loaded from that page. You might want to shave this page down somewhat by dropping some of the response headers. Or if you really want to (not) send a message, you could use the shield vmod, which enables Varnish to discard a connection in VCL, sending a TCP reset.

Or, if you really want to, you could pair this with firewall rules to discard the reset so the client wouldn’t know and just hang there without getting any notification of connection termination. This could starve the attacking clients resources and reduce the intensity of the attack. Look at std.set_ip_tos to see how one could mark a connection in a way that could be picked up by the firewall.

Other options include gathering the ip addresses of the attackers and feeding them to ipset or a similar black hole. I've heard good things about ipset lately, it is supposed to be very robust.

A blast from the past - SYN floods

SYN floods are back and they are wrecking havoc. Last year, here in Norway a small number of people managed to keep some rather prominent sites down for quite some time. They used botnet-powered SYN floods. According to Audun Ytterdal of VG Nett RHEL and Ubuntu won’t cope too well with big SYN floods. VG has a couple of high end load balancers and they managed to keep up but they do have their limits. And at over €100K a pop they are an expensive thing to scale out. Especially when the reason is that some little script kiddie somewhere wants to play a prank on you.

I asked on Google+ if anyone would have any advice on how to withstand a SYN flood with off the shelf hardware and Chris Davis responded. The SYN cookie support in Linux is good, but there could probably be some improvement. His recommendations is to upgrade the kernel beyond 3.2. Futhermore you need to make sure that the hardware works properly, interrupt line sharing, polling and all. Go dig around in the networks card source code and experiment a bit.

To activate and tune the SYN cookie support you should deal with these three kernel parameters:

tcp_syncookies - whether or not to send cookies when needed.

tcp_max_syn_backlog - how many connection we’ll have outstanding before we assume we’re under a flood.

tcp_synack_retries - how many tries should we retry to respond on a connection request before giving up. Setting it to 3 will make the kernel give up the connection rather quickly (45 seconds or so).

Most of the attack will usually come from a handfull of hosts. The rest is usually comming from hosts with bad connectivity and can therefore be more or less ignored. So, if you’re able to identify the top 10 hosts and block these chances are most of the attack will be lifted. On newer Linux kernels ipset is now a part of the official kernel.

There are of course alternatives to this DIY way of doing things. There are whole companies dedicated to fighting DDoS attacks. On a personal level I would like to see much more community provided information on how to fight DDOS attacks so those of us who are subject to these attacks can fight them using as few resources and money as possible. If you know of any such website where information is gathered, please leave a comment. And good luck.

Picture: some rights reserved girolame.