The A/B Test

In simple terms, it is an experimental method that compares two variants – A and B – with the objective of determining which variants produces the best results.

Also known as split testing, A/B tests consist of dividing an audience into two or more groups (depending on how many variants are being compared) and presenting each group with versions of the same impulse or content. The difference between the versions usually lies in one variable, which allows the tester to isolate the cause of the difference in results.

The image below shows an example of an A/B test ran by Amazon on the day that this post was written. The sole difference between the two versions of the homepage presented is the second image of the carousel.

Almost anything can be subjected to A/B testing. Websites, emails, ads, and even app interfaces are frequently tested. Besides, there are tons of different variables that can be manipulated to generate variants for the test. Some of the most common variables are:

• Headline
• Call to action text
• Call to action color
• Call to action location
• Images
• Subject lines (for emails)
• Ad copy

Before you decide what to test, first formulate a hypothesis of how these variables can affect your results. This will allow you to make a decision of which variable should be tested first.

The duration of the test can vary depending on how many versions are being tested at the same time. The amount of traffic you are receiving is also an important metric. You want to allow the test to run for enough time so that each version is viewed by enough people. This will enable you to extract unskewed conclusions. However, you should also avoid running the test for too long. Results could be influenced by too many outside impulses over which you have no control.

A/B testing can help a product manager to improve results, from top of the funnel (as click through rate or bounce rate) to lower funnel outcomes (such as revenues).