UPDATE: mss noted on Github that I manipulted the Set-Cookie header as if it were the Cookie header, which is wrong. This article has been corrected accordingly.

You may have read Dridi's blog post about cookies, and realized that he abhors them. So, in a playful attempt to piss him off, I decided to write Yet Another Post™ about cookies!

I know, I'm a terrible colleague, but, you, dear reader will benefit from this terribleness (I hope), as this article aims to explain in detail how we can use cookies to provide sticky sessions through Varnish. So, let's jump in the fire!

Flattery is here to stay

It's said that "Imitation is the sincerest form of flattery", and if that's true, this blog post is going to be all about flattering HAProxy. Often, I started writing because someone asked "Hey! Can you do that with Varnish?". That being answering this stackoverflow question: how can you consistently send the requests of the same HTTP "session" to the same backend. Taken mostly from the the HAProxy manual, the answer is succinct, clear, and demonstrates how HAProxy is well-suited for the task at hand.

So, obviously, I got jealous, and set out to write appropriate VCL, then thought it would be nice to cover all three cases, then that it would be nice to let the world know. And here we are!

In this piece, we'll use Varnish as a pure load balancer, totally ignoring how awesome it is at caching, and focusing solely on correctly routing the first request and all the subsequent ones to the same backend. But know that we can do both if needed. Also, since we'll manipulate a lot of cookies, we'll make heavy use of the vmod-cookie (part of varnish-modules) to avoid dreary regex.

Some kind of cookie monster

The very first option HAProxy gives you is to create an extra cookie for your routing: it's simple but smart. The VCL corresponding to that option looks like this:

import cookie;
import directors;
import std;
import header;

sub vcl_recv {
    cookie.parse(req.http.cookie);
    set req.http.server = cookie.get("id");

    if (req.http.server == "s1") {
        set req.backend_hint = s1;
    } else if (req.http.server == "s2") {
        set req.backend_hint = s2;
    } else {
        if (std.rand(0, 100) < 50) {
            req.backend_hint = s1;
        } else {
            req.backend_hint = s2;
        }
    }
    return (pass);
}

sub vcl_deliver {
    header.append(resp.http.set-cookie, "id=" + req.http.server);
}

The concept is twofold, and very straightforward:

• when we receive a request (in vcl_recv), we set a cookie if none was present, then assign the backend based on that.
• when we deliver, we inject the cookie, so that the user will come back with it for the next request. This means that we'll override the cookie with the same value most of the time, but it's really not a problem.

Elegant and quick, I like it. But you may not want to add that extra cookie to the mix...

Devil's dance

HAProxy has a solution for when you refuse to create a new header: you can prefix an existing one. Very simply, if your backend replies with "set-cookie: JSESSIONID=0123456789", HAProxy automatically adds the name of the backend (s1 for example) to it before sending it to the client that will see "set-cookie: JSESSIONID=s1~0123456789". Obviously, it's necessary to remove it again before sending it to the backend.

Varnish can do this easily using only a tiny bit of regex:

import cookie;
import header;

sub vcl_backend_response {
    # if there's a cookie JSESSIONID, we prefix it with the backend
    set beresp.http.x-tmp = header.get(beresp.http.set-cookie,"JSESSIONID=");
    if (beresp.http.x-tmp != "") {
        set beresp.http.x-tmp = regsub(beresp.http.x-tmp, "=", "=" + beresp.backend + "~");
        header.remove(beresp.http.set-cookie, "JSESSIONID=");
        header.append(beresp.http.set-cookie, beresp.http.x-tmp);
    }
    unset beresp.http.x-tmp;
}

sub vcl_recv {
    cookie.parse(req.http.cookie);
    # remove the prefix (if any), and save it as a header
    set req.http.prefix = regsub(cookie.get("JSESSIONID"), "~.*", "");
    cookie.set("JSESSIONID", regsub(cookie.get("JSESSIONID"), "^[^~]+~", ""));
    set req.http.cookie = cookie.get_string();

    # use the saved header to select the backend
    if (req.http.prefix == "s1") {
        set req.backend_hint = s1;
    } else if (req.http.prefix == "s2") {
        set req.backend_hint = s2;
    } else {
        if (std.rand(0, 100) < 50) {
            req.backend_hint = s1;
        } else {
            req.backend_hint = s2;
        }
    }
    return (pass);
}

There's nothing complicated in there once you learn the steps of this little dance: parse the cookie, add/remove the backend, set the new cookie string. The real trick is the regex "^[^~]+~" describing the "starting string with a number of non-tilde characters, then a tilde". If this seems weird to you, I heartily recommend that you bookmark regex101.com as it has saved me countless hours.

The corresponding text case is here.

Don't tread on me

This is fine and all, but as cookies are supposed to be an opaque handle that you're not supposed to touch, maybe you feel that all this is a bit invasive? HAProxy got you covered! They have a third option: the stick-table. This feature will retain the cookies it sees as well as what backend issued them, and will route accordingly.

That's something we can do with Varnish too, obviously, but we'll need an extra vmod: kvstore. You can think of it as vmod-var on steroids (kids, don't do drugs; drugs are bad!), and among the extra things it can do is assigning TTL to your keys, so that you only store stuff that is actually used. With vmod-var, we would risk growing our store without any limit, and that would be bad.

Here's the code:

import cookie;
import directors;
import std;
import kvstore;

sub vcl_init {
    # init the store
    kvstore.init(0, 1000);
}

sub vcl_backend_response {
    # only grab the JSESSIONID=VALUE part of the cookie
    set beresp.http.x-tmp = regsub(header.get(beresp.http.set-cookie,"JSESSIONID=", " *;.*", "");
    # if there's a cookie, use it as key to remember the backend
    # for 15 minutes
    if (beresp.http.x-tmp != "") {
        kvstore.set(0, cookie.get("id"), beresp.backend, 15m);
    }
    unset beresp.http.x-tmp;
}

sub vcl_recv {
    cookie.parse(req.http.cookie);
    
    # grab the corresponding server, or "none
    set req.http.server = kvstore.get(0, cookie.get("id"), "none");

    if (req.http.server == "s1") {
        set req.backend_hint = s1;
    } else if (req.http.server == "s2") {
        set req.backend_hint = s2;
    }  else {
        if (std.rand(0, 100) < 50) {
            req.backend_hint = s1;
        } else {
            req.backend_hint = s2;
        }
    }
    return (pass);
}

All within your hands

For once, this was a short post, right? Of course, we didn't dig too deep into specific problems, such as obfuscating the backend name (to avoid cookie forgery), or leveraging vmod-stendhal to avoid if-else'ing on req.http.prefix, but we clearly showed that the VCL and its vmods provide us with the necessary building blocks to build whatever policy we need regarding this subject.

If you're ready to learn more about how to administer and master your own Varnish instance, check out the training sessions we have coming up.

Photo (c) 2009 Watashiwani used under Creative Commons license.