Using Varnish to assist with AB Testing

While working with a recent client project, they mentioned AB Testing a few designs. While I enjoy statistics, we looked at Google’s Website Optimizer to track trials and conversions. After some internal testing, we opted to use Funnels and Goals rather than the AB or Multivariate test. I had little control over the origin server, but I did have control over the front-end cache.

Our situation reminded me of a situation I encountered years ago. A client had an inhouse web designer and a subcontracted web designer. I felt the subcontracted web designer’s design would convert better. The client wasn’t completely convinced, but agreed to running two designs head to head. However, their implementation of the test biased the results.

What went wrong?

Each design was run for a week, in series. While this provided ample time for gathering data, the inhouse designer’s design ran during a national holiday with a three day weekend, and the subcontractor’s design ran the following week. Internet traffic patterns, the holiday weekend, weather, sporting events, TV/Movie premieres, etc. added so many variables which should have invalidated the results.

Since Google’s AB Testing has session persistence and splits traffic between the AB tests, we need to emulate this behavior. When people run AB tests in series rather than parallel, or, switch pages with a cron job or some other automated method, I cringe. A test at 5pm EST and 6pm EST will yield different results. At 5pm EST, your target audience could be driving home from work. At 6pm EST they could be sitting down for dinner.

How can Varnish help?

If we allow Varnish to select the landing page/offer page outside the origin server’s control, we can run both tests run at the same time. An internet logjam in Seattle, WA would affect both tests evenly. Likewise, a national or worldwide event would affect both tests equally. Now that we know how to make sure the AB Test is fairly balanced, we have to implement it.

Redirection sometimes plays havoc on browsers and spiders, so, we’ll rewrite the URL within Varnish using some Inline C and VCL. Google uses javascript and a document.location call to send some visitors to the B/alternate page. Users that have javascript disabled, will only see the Primary page.

Our Varnish config file contains the following:

sub vcl_recv {
  if (req.url == "/") {
      char buff[5];
      sprintf(buff,"%d",rand()%2 + 1);
      VRT_SetHdr(sp, HDR_REQ, "\011X-ABtest:", buff, vrt_magic_string_end );
    set req.url = "/" req.http.X-ABtest "/" req.url;

We’ve placed our landing pages in /1/ and /2/ directories on our origin server. The only page Varnish intercepts is the index page at the root of the site. Varnish randomly chooses to serve the index.html page from /1/ or /2/, internally rewrites our URL and serves it from the cache or the origin server. Since the URL rewriting is done within vcl_recv, subsequent requests for the page don’t hit the origin. The same method can be used to test landing pages that aren’t at the root of your site by modifying the if (req.url == “”) { condition.

You can test multipage offers by placing additional pages within the /1/ and /2/ directories on your origin along with the signup form. Unlike Google’s AB Test, Varnish does not support session persistence. Reloading the root page will result in the surfer alternating between both test pages. Subsequent pages need to be loaded from /1/ or /2/ based on which landing page was selected.

When doing any AB Test, change as few variables as possible, document the changes, and analyze the difference between the results. Running at least 1000 views of each is an absolute minimum. While Google’s Multivariate test provides a lot more options, a simple AB test between two pages or site tours can give some insight into what works rather easily.

If you cannot use Google’s AB Test or the Multivariate Test, using their Funnels and Goals tool will still allow you to do AB Testing.

Tags: , ,

5 Responses to “Using Varnish to assist with AB Testing”

  1. Scalability links for Feb 28th 2010 | Scalable web architectures Says:

    […] Using Varnish to assist with AB testing – Testing new uncooked features by external customers get more difficult as products become more mature and stable. Tools like “varnish” could be used to test different pages/features. […]

  2. uberVU - social comments Says:

    Social comments and analytics for this post…

    This post was mentioned on Twitter by mcd34: blog post: Using Varnish to assist with AB Testing #abtest #analytics #Varnish…

  3. perbu Says:

    I’ve been thinking about this a bit and I don’t see why we can’t do persistent AB testing with Varnish. If you’re set a cookie “abtest” to “A” or “B” and then pick up the cookie in vcl_recv and either rewrite the URL (prepending /1 or /2) or set a request header to the backend based on that cookie you could easily accomplish this. If you set a synthetic request header you could also Vary on this header increasing the cache efficiency.

  4. cd34 Says:

    Where do you set the cookie? Since most AB Tests are testing landing pages, usually it is the first time we’re seeing that particular surfer. So, the decision needs to be made within Varnish. Since we don’t see the request hit on the backend, we have a small webbug that reports which page was served for analysis.

    The only method I could think of to make and cache that decision within Varnish was to use inline C since I didn’t have access to a random function in VCL.

  5. cd34 Says:

    Modified to add vrt_magic_string_end as per Christopher Cato’s debugging and DocWilco in IRC

Leave a Reply

You must be logged in to post a comment.

Entries (RSS) and Comments (RSS).
Cluster host: li