Data Analytics: Foundations to Practice · A/B Testing and Experimentation
Designing a Valid A/B Test
A test's validity is largely determined before it even launches, through a handful of specific design decisions that are far easier to get right upfront than to fix after data collection has already begun.
A well-designed test defines a single, specific primary metric it's meant to move before the test actually launches, rather than launching first and then searching through many different metrics afterward to find one that happens to show a statistically significant result, a practice that runs directly into the multiple comparisons problem covered in Module 7, since checking enough different metrics after the fact makes finding at least one that looks significant purely by chance increasingly likely.
- Defining a single primary metric before launch avoids the multiple comparisons problem of searching many metrics afterward for a significant-looking result.
- Genuine randomization requires a truly random assignment process, since convenient methods like alternating sequential order can introduce subtle systematic bias.
- Calculating adequate sample size before launch prevents ending too early and prevents the false-positive-inflating practice of stopping as soon as results look significant.
- Contamination between control and treatment groups, especially in social features, can cause the treatment's effect to spill over and understate its true impact.