What is statistical significance in an A/B test
An A/B test result is statistically significant when the gap between versions is so large that it would be very rare to see it by pure chance if both actually performed the same. The usual bar is 95%, meaning a p-value below 0.05.
By Fernando Blanco Dosil · Updated November 2, 2026
The p-value, without jargon
The p-value answers this question: if the variant and the original converted exactly the same, how likely would I be to see a difference as big as this one, or bigger? If that chance is very low (under 5%), you conclude the most reasonable explanation is that the difference is real.
What the p-value does not tell you, even though many people read it that way, is the probability that the variant is better. A p-value of 0.03 does not mean a 97% chance of winning. It is a different question, and that confusion is behind a lot of bad decisions.
In Spanish you will see both significación and significancia. They mean the same thing.
The most common mistakes
The classic method works well when used as designed: you decide the sample in advance, launch the test and look at the result once, at the end. Almost nobody does that.
- Checking every day and stopping when it turns significant. If you look daily, sooner or later the p-value will cross 0.05 by chance. With many looks, the false winner rate climbs far above 5%.
- Stopping early. A variant that wins the first weekend of a sale can lose the rest of the month. Without full weeks, the result does not represent your visitors.
- Testing many variants at once. With five variants against the original, the chance that one comes out significant by luck is no longer 5%, but quite a bit more.
- Watching many metrics and keeping the one that wins. Measure twenty things and one will come out significant even if the change does nothing.
Classic significance versus probability to be best
Two ways to decide a test. They answer different questions.
| p-value (frequentist) | Probability to be best (Bayesian) | |
|---|---|---|
| Question it answers | How unusual this result would be if there were no difference | How likely this version is to be the best |
| How you read it | Below 0.05, you rule out that they are equal | 96% means a 96% chance it wins, no translation |
| Checking daily | Inflates false winners | Tolerated better, with minimum sample and duration |
| What else it tells you | Nothing about how much you risk | How much you would lose if you pick wrong (expected loss) |
How Bixlow handles it
Bixlow does not report p-value results. It uses Bayesian statistics and shows the probability that each version is the best, the estimated lift with a credible interval and the expected loss. It only declares a winner when the probability reaches 95%, each version has the minimum sample and the test has run for the minimum duration, so checking results every day does not lead you to stop on a spike.