# What is statistical significance in an A/B test

> An A/B test result is statistically significant when the gap between versions is so large that it would be very rare to see it by pure chance if both actually performed the same. The usual bar is 95%, meaning a p-value below 0.05.

- Source: https://bixlow.com/en/glossary/statistical-significance
- Language: en
- Updated: 2026-11-02
- Publisher: Bixlow

## The p-value, without jargon

The p-value answers this question: if the variant and the original converted exactly the same, how likely would I be to see a difference as big as this one, or bigger? If that chance is very low (under 5%), you conclude the most reasonable explanation is that the difference is real.

What the p-value does not tell you, even though many people read it that way, is the probability that the variant is better. A p-value of 0.03 does not mean a 97% chance of winning. It is a different question, and that confusion is behind a lot of bad decisions.

In Spanish you will see both significación and significancia. They mean the same thing.


## The most common mistakes

The classic method works well when used as designed: you decide the sample in advance, launch the test and look at the result once, at the end. Almost nobody does that.

- Checking every day and stopping when it turns significant. If you look daily, sooner or later the p-value will cross 0.05 by chance. With many looks, the false winner rate climbs far above 5%.
- Stopping early. A variant that wins the first weekend of a sale can lose the rest of the month. Without full weeks, the result does not represent your visitors.
- Testing many variants at once. With five variants against the original, the chance that one comes out significant by luck is no longer 5%, but quite a bit more.
- Watching many metrics and keeping the one that wins. Measure twenty things and one will come out significant even if the change does nothing.


> **Example.** You launch a test on your product page. On Wednesday the p-value is 0.04 and the variant is 18% ahead. You stop and publish. A month later sales are flat. It wasn't bad luck: you stopped on a spike of noise. Had you waited for the planned sample, that 18% would most likely have faded to nothing.


## Classic significance versus probability to be best

Two ways to decide a test. They answer different questions.

|  | p-value (frequentist) | Probability to be best (Bayesian) |
| --- | --- | --- |
| Question it answers | How unusual this result would be if there were no difference | How likely this version is to be the best |
| How you read it | Below 0.05, you rule out that they are equal | 96% means a 96% chance it wins, no translation |
| Checking daily | Inflates false winners | Tolerated better, with minimum sample and duration |
| What else it tells you | Nothing about how much you risk | How much you would lose if you pick wrong (expected loss) |


## How Bixlow handles it

Bixlow does not report p-value results. It uses Bayesian statistics and shows the probability that each version is the best, the estimated lift with a credible interval and the expected loss. It only declares a winner when the probability reaches 95%, each version has the minimum sample and the test has run for the minimum duration, so checking results every day does not lead you to stop on a spike.


## FAQ

### Does 95% significance mean a 95% chance of winning?

No. It means that, if there were no difference, you would see a result like this less than 5% of the time. The chance of winning is a different calculation, the one Bayesian statistics does.

### Why 95% and not another number?

It is a convention. Some teams use 90% for low-risk decisions and 99% for changes that are expensive to undo.

### Can I look at results while the test is running?

Looking is not the problem; stopping as soon as you see a nice number is. Set the sample and duration beforehand, and don't decide before you meet them.

### Is statistical significance the same as confidence level?

They are two sides of the same threshold. A 95% confidence level means you call a result significant when the p-value is below 0.05.

## Further reading

- [What is Bayesian A/B testing](https://bixlow.com/en/glossary/bayesian-ab-testing)
- [What is sample size](https://bixlow.com/en/glossary/sample-size)
- [Bixlow's statistics](https://bixlow.com/en/platform/statistics)