A/B testing or Multivariate testing?

Isn't it cool to even have the choice to do A/B testing or Multivariate Testing courtesy of Google Website Optimizer?

I have been using both tools extensively over the past several months. I am surprised that many online merchants haven't tried doing website optimization this way, although once they "get it", they'll want to run experiments all the time.

And why wouldn't you? I find it very satisfying to let customers make the decision as to what works and what doesn't, instead of the personal preferences a website owner or opinionated web designer. Don't redesign your site just for the sake of it, or be told by someone that it's time to get a "fresh look". Some of the best converting sites are not pretty. At the very least, make sure you capture "before and after" data in your analytics solution.

A re-design by definition changes so many elements that you won't know what specific elements contribute to your conversion rate. Is it the Hackersafe logo in the left nav or the Free Shipping message in the top nav? This is akin to doing an A/B test: you compare one page vs another and see what converts better. However, you won't know what element contributes most.

For this reason, I prefer multivariate testing. Here, you essentially compare different elements on the same page. So you would test the Hackersafe element and Free Shipping element on the same page. The experiment will then show you what element contributes the most. You can even find out if a particular combination of elements converts better than another.

But there is another reason I prefer multivariate testing over AB testing, which I don't think gets talked about much. Perhaps it's not too huge an issue in practice, but as a heavy user of web analytics, I care a lot about getting good data all the time.

In an A/B test, if your A page is an important landing page, i.e. the first point of entry of your site visitors, consider this: the experiment will keep roughly 50% of visitors on A whereas the other 50% will get re-directed to the B page, as it should be. But, what happens to the referrer information? If the visitor sees page A the original referrer (e.g. Google organic) will be preserved. However, if the visitor lands on page B, analytics will think that the visitor came directly to the site. So, the original referrer information is lost.

Does this make sense? Anybody in the A/B camp disagree with me?

Comments

[gretchen]

i think wso appends the "a" referrer info to the "b" page so you can still see the referrer info.

[Stephen Dean]

I'm not sure about Google's software. I have my own split test software at www.easysplittest.com. It uses an index.php page and calls page A and B from .htm docs. So the referrer information will always be the same.

As far as when to use Split testing or Multivariate testing. If you're getting a ton of traffic I'd always go with Multivariate. But if I need to find out something fast and the traffic's only trickling in, then I go for the split test.

[AndyEd]

Losing referrer data is no fun, for sure, but there's no real reason why this needs to be a consequence of an A/B split. Using a touch more server side goodness can allow landing A and landing B to be served directly, and you'd tag your users with _utmSetVar if using Google Analtyics. Alternatively, you could make all your page changes with client side script even in the A/B case.

MV methods do give greater insights into the driving factors behind a test, but a simple restrained A/B that tests the just one factor can do that for you too.

My favorite aspect of MV is getting more power to detect differences by testing more stuff per user and throwing some stats in to separate it all out.

[Michael Whitaker]

@gretchen

Thanks for commenting. What I am seeing is that if a visitor arrives on A for the very first time, i.e. no cookies are set in the visitor's browser, website optimizer will redirect to B before GA runs. So, when GA runs on B, it will see the referrer as coming from A. This seems actually like the only plausible scenario and is in fact mentioned in GA's help files: http://www.google.com/support/analytics/bin/answer.py?hl=en&answer=55477

But I will gladly stand corrected if I am not seeing this correctly!

Michael

[Greg]

Actually WSO retains the referrer info and appends it to the B page as part of the redirection.

[The Magic Of Testing]

Hey Michael!

I was actually relieved to see Greg's comment above, b/c I thought that's how it worked, but reading your post, I got a bit concerned.

Side note: I'm 100% in the multivariate camp, (caveat:) if you've got the traffic to justify it.

Thanks for posting this though. It was definitely interesting food for thought.

Warmest,

Jonathan Kraft
http://www.TheMagicOfTesting.com

[Suhail]

Cool article, definitely agree with the value of A/B testing.

We just recently pumped out hopefully a way better solution for this type of stuff at http://mixpanel.com

There's also an article written up on how to do A/B testing with mixpanel:
http://blog.mixpanel.com/2009/01/ab-testing-your-website-with-mixpanel/

[Keith Holloway]

There's a great thread on this problem here:
http://www.google.com/support/forum/p/websiteoptimizer/thread?tid=648e25778bb6aee1&hl=en

You can read through yourself, but here is my summary:

1) A really simple solution by Shoretel almost solves the problem, but may not work consistently:

"In order to preserve the Traffic Source information for GA, you need to place your GA tracking code above your GWO control code at the top of your page. That way Google Analytics will be able set it's traffic source cookie before the visitor is redirected to your "B" page therefore preserving their traffic source information."

It works in theory, but the problem with the above approach is that the redirect may cancel the tracking request if it's not fulfilled before the redirect happens.

2) A better solution from Prusak.com writes the original referrer to a cookie, and then picks it up again on the test page:

Original Solution:
http://www.prusak.com/google-analytics-referrer-override/

Then updated here:
http://www.prusak.com/google-ananlytics-initial-referrer-update/