A/B testing
An experiment in which people are randomly split between two or more versions of a product and a metric is compared to see which version works better.
What it’s for
It tells you whether a change really improves something, not just whether the team thinks it does.
Key elements
At random
Random assignment makes the version the only difference between the groups.
The metric first
Decide what you will measure before starting, not after seeing the results.
Enough people
With little data, differences may be due to chance.
How it’s done
Define the change, the metric and how many people you need.
Assign people at random to the versions.
Compare the metric and decide.
Applications in design
When to use it
To choose between two versions of a screen, a message or a price when you have a lot of traffic.
Avoid
Stopping the experiment as soon as you like the numbers: the result may be chance.
Don’t fall for it
It tells you what works better, not why: combine it with interviews or usability tests.
Connections
Authors and works
- 2020
Ron Kohavi, Diane Tang and Ya Xu
They brought together in a practical guide how to design and analyze controlled experiments in digital products.
Sources
Original sources
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
Last reviewed: