Controlled Testing: the Champion-Challenger Approach

Controlled Testing: the Champion-Challenger Approach

Controlled testing is one of the most valuable tools in your marketing repertoire. Controlled testing takes you from a place of “guessing” at your marketing message and design to a place where you can base your website development and marketing on actual data. See how your customers react, and whether a specific marketing message or page design works better with your customer base. Today, we’re going to look at a scientific method of controlled testing: the champion-challenger approach.

Start with a Control, and Test Challengers

To accurately test a website, you need to start with a control – the current, unaltered website – and test variants against it concurrently to determine which page functions the best. In a controlled test, it’s important to keep factors as constant as possible to ensure that your results aren’t polluted.

For example, if you test one website variant by serving only that site to all of your viewers for a week, you’re not getting an accurate, scientific test. What if something changes outside your site that results in a spike in traffic that week – like someone posts a link on social media or you get featured somewhere? The results you see won’t be because of your site changes – they’ll be dependent on some external driver.

The only way to get accurate, scientific results of your development efforts is to serve both your control site and your variant at the same time. In an A/B/n test, also sometimes called a split test, you randomly assign every visitor to your site to either the control or one of your variants. Visitor A may get your control, while visitor B 10 seconds later may get your third variant, and visitor C a minute later might get your first variant.

The idea is that you’re evenly distributing your visitors across the control and your variants – concurrently and randomly – so that you’re getting a more accurate representation of what’s actually going on within your site, without being polluted by outside factors. This is the champion-challenger approach.

Using cookies, the visitors will see the same page variation on each subsequent visit to your site. This keeps the user experience consistent, and provides a seamless, valid test.

Once the visitor takes your desired action – whether that’s signing up for a mailing list, purchasing a product or something else entirely – the test records a conversion, and the result is tallied for the visitor’s assigned variant.

A good testing tool will then calculate which page delivers the highest conversion rate – whether it’s the control or one of the variants – and can determine whether the test results are statistically significant – i.e. whether they reflect a significant enough sample size to be a conclusive indicator. If you see positive results that are statistically significant, you know which variant is king!

The Importance of a Good Testing Tool

To conduct a controlled, valid split test, you need a good testing tool. Google AdWords, for example, isn’t a controlled testing tool – it provides skewed data that can easily mislead you. You need a tool designed for controlled website testing, and you need to run the experiment long enough to achieve the statistical significance that can tell you whether you’ve tested “enough” people to take the results as valid.

A good testing tool will:

  • Split traffic evenly, and randomly, among control and variations
  • Use a cookie to ensure the visitor is served the same site every time they come
  • Track a conversion goal for each variant page
  • Give you reports that show the conversion rate for each variant
  • Give you statistical significance so you can decide when a test is complete and when you can declare a page the “winner”

Controlled testing with a good tool is extremely important if you want to boost your marketing success. The wrong tool can provide skewed or misleading results. Approach a controlled test scientifically, or you’re wasting your money and development effort

Written by admin



Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *