Balancing profit and learning in A/B testing
A/B testing, or split testing, is commonly used in web marketing to decide which of two design options performs better. If you have so many visitors to a site that the number of visitors used in a test is negligible, conventional randomization schemes are the way to go. They're simple and effective.
But if you have less traffic so that the number of visitors involved in a test is appreciable, you might be concerned with possible lost revenue during the test itself. The point of A/B testing is to improve profitability after the test, not during the test. If you also want to consider profitability during the test, you might want to consider more alternatives.
My experience with testing comes from a context where the stakes are higher than improving conversion on web sites: treating cancer patients. You want to find out which treatments performed better for the sake of future patients, those who were treated after the randomized trial. But you also want to treat the participants in the clinical trial effectively. Two ways we would do that are early stopping rules and adaptive randomization. Both practices are applicable to A/B testing web pages.
A conventional clinical trial might take a few hundred patients and randomize half to one treatment and half to another. But if one treatment appears to be much more effective, at some point it becomes unconscionable to keep assigning the less effective treatment. So you stop the experiment early. You might want to do the same with web designs. If you planned to show two variations of a page to 500 visitors each, but after 100 visitors it's obvious which version is performing better, you'd like to stop the test and show everyone the better page. On the other hand, if you have so many visitors that you're not concerned with what happens to the 1000 visitors in the test, just let the test run to completion.
Another approach is to compromise between equal randomization and early stopping. Suppose A is performing better than B, but not so much better that you're willing to stop and declare A the winner. You might keep randomizing, but increase the probability that the test will assign A. If A really is better, more visitors will see the better page. But if you're wrong and B is really better, you may still discover this because some visitors are still seeing B. If B keeps performing better, the tide will turn and the test will prefer it. This is called adaptive randomization. The more evidence there is that one version is better, the higher the probability that you'll show people that version.
One way to use adaptive randomization is variable experiment sizes. Instead of deciding a test size in advance, you test until you're satisfied that you've found a winner. That may require fewer visitors than a conventional A/B test. It may also require more, but only when there's a good reason to. The test may go into overtime, so to speak, because the two versions are performing similarly, in which case you'd like to keep testing longer to find which is better.
It's easy to fall into thinking that the winner of a test will be used forever, whether you're testing web pages or cancer treatments. But this isn't the case. The winner will eventually be tested against something else, maybe very soon. This means that you might want to put a little more emphasis on the performance during the test and not just performance after the test, because there may not be much opportunity for performance after the test.
If you'd like to discuss how adaptive randomization could benefit your testing, please let me know.