Live AB testing with Varnish and VCS

In Varnish Software, we tend to be building tools more than solutions. I like tools that are open and flexible. When we designed Varnish Custom Statistics we wanted to make sure that VCS would be such a tool, something that be useful for everyone, allowing for great flexibility and power.

VCS is a timeseries database that takes input from Varnish. You’ll feed it data from your VCL and it starts recording data. You decide what the keys are and it does the rest.
One useful thing is to use this to conduct AB tests. Let’s see how we can build a setup that let’s you do AB-tests with Varnish and VCS.

Step 1: Assigning a user to the A or B group

There are multiple ways of doing this. You could let the traffic hit the backend and have it set a cookie - but that’s no fun. We have VCL and we want this to happen in Varnish.

In vcl_recv we separate out traffic that hits our landing page - /foo and then we set a cookie. To do the actual cookie manipulation we use the cookie VMOD. We could do it with regular expressions but that code would be pretty bad looking. So, in this post I’ll use the excellent cookie VMOD.

# Unset the client header to make sure it is clean.
unset req.http.abgroup;
if (req.url == “/foo”) {
  if (std.random(0, 100) < 50) {
    # A group
    set req.http.abgroup = “A”;
 } else {
   # B group
    set req.http.abgroup = “B”;
}

Then in deliver we persist the group through a cookie. We make the cookie last 60 minutes.

if (req.http.abgroup) {
      set resp.http.Set-Cookie = "abgroup=" + req.http.abgroup + "; Expires=" + cookie.format_rfc1123(now, 60m) + "; httpOnly";
}

Step 2: Separating out the A and B groups and serving different content

The next step would be to make sure the different groups get different content. The core idea here is to create a synthetic client header and to rely on Vary to force Varnish to serve up different content.

We already set the abgroup request header for new requests. We just need to make sure it is set on returning clients.

if (req.url == “/foo”) {
  if (cookie.get(“abgroup”)) {
  set req.http.abgroup = cookie.get(“abgroup”);
} else {
  # No group is set. Set it.
  if (std.random(0, 99) < 50) {
  # as before
  }
}

Step 3: Changing the backend to issue to right version

Varnish will then keep two distinct versions of /foo in cache. One for A and one for B.

Step 4: Tracking traffic generated from the A and B group

Now we’ve assigned the cookie abgroup to the test. To count the number of transactions in each group we push the data off to VCS:

   std.log(“vcs-key: abgroup:” + req.http.abgroup);

It might make sense to add a unique session ID to this so we also get a count of each unique users in each group. To do this we must have a session cookie or similar. VCS will then start counting uniques in each group. In this example we’ll just reuse the Google Analytics session cookie __utma.

First we’ll stick in a separate header:

set req.http.session-id = cookie.get(“__utma”);

Then add the session ID to the transaction:

std.log("vcs-unique-id: " + req.http.session-id);

Step 5: Identifying a conversion and tracking it

Our experiment is now underway. Now we need to get data from the test to measure conversion rate on each user. Let’s say that we decide that a successful conversion is one where the use submits a certain form. Let’s set up from VCL code in vcl-recv to intercept it:

if (req.url == “/foo/form” && req.method = “POST) {
    std.log(“vcs-key: abconv:” + req.http.abgroup);
}

And you’re done. You can now query VCS for live data on the conversion. You'll get the number of transactions in each group, the number of people in each group and the number of successful conversions.

VCS requires a Varnish Plus subscription. If you want to you can give VCS a spin. Let us know if you're considering a Varnish Plus trial.

Image (c) 2008 Horia Varlan used under Creative Commons license.