Product Management: Foundations to Practice · Product Analytics and Metrics
A/B Testing Fundamentals
A/B testing is simple in concept and easy to run badly. This chapter covers the statistical discipline required to trust a test result at all.
A/B testing randomly splits users into two or more groups, exposing each to a different version of a product experience, then compares a predefined success metric between groups to determine which version performs better. Randomization is the critical, easy-to-overlook requirement: if the groups are not genuinely randomly assigned, any observed difference could reflect a pre-existing difference between the groups rather than the effect of the change being tested.
Key Takeaways
- Genuine randomization is the critical, easy-to-overlook requirement for a trustworthy A/B test.
- Statistical significance requires sufficient sample size and duration; low-traffic decisions may not be suited to A/B testing at all.
- Peeking at results early and stopping when favorable produces false positives; pre-commit to a sample size and duration.
- Defining the success metric before launching a test guards against p-hacking, where testing many metrics eventually finds a false positive by chance.