Statistical significance explained without any maths

You don't need to compute a p-value. You do need to know what one claims, because otherwise you'll accept results that don't support the decisions you make with them.

The question it answers#

Imagine your variant is identical to control — literally the same page, labelled differently. Run that "experiment" and you will still see a difference between the two groups, because different people landed in each and people vary. That difference is pure chance.

Significance answers one narrow question: if there were genuinely no difference, how often would chance alone produce a gap at least as big as the one I'm seeing?

When a tool says 95% confidence, it means: a gap this large would come up by chance less than 5% of the time. That's it. It's a statement about how surprising your data would be in a world where the change did nothing.

What it does not tell you#

It doesn't tell you the probability that your variant is better. This is the most common misreading. "95% confident" is not "95% chance B wins". It's a statement about the data under an assumption, not about the truth of the hypothesis.

It doesn't tell you the effect is big. With enough traffic, a 0.3% improvement becomes significant. Significant means detectable, not important. Always read the size alongside the significance.

It doesn't tell you the effect is stable. A result significant in November tells you about November's traffic, campaigns and customers.

And it means nothing at all if you peeked. This is the one that actually ruins results in practice. The 5% guarantee assumes you looked once, at a pre-declared sample size. If you check daily and stop when it goes green, your real false-positive rate is far higher — commonly 20–30%. You haven't run a 95% test; you've run a procedure that finds "winners" in noise.

Read the interval, not the verdict#

The single most useful habit is ignoring the green tick and reading the confidence interval — the range the true effect plausibly sits in.

"+7%, interval +2% to +12%" says the change probably helped, somewhere between a little and a lot. You can act on that.

"+7%, interval −4% to +19%" contains zero. The true effect might be negative. The headline number is the same and the conclusion is completely different.

"+0.4%, interval +0.1% to +0.7%" is statistically significant and commercially irrelevant. Ship it if it's free, but don't build a strategy on it.

Bixlow leads with the interval rather than the verdict for exactly this reason. A single number invites a yes/no reading; a range forces you to think about magnitude and uncertainty at the same time.

The four sentences#

When a result lands, say these out loud:

1. The observed difference is X%.
2. The plausible range is A% to B%.
3. Does that range include zero? If yes, you don't have evidence of a difference — you have an inconclusive test, which is not the same as evidence of no difference.
4. Is the bottom of the range big enough to matter? If even the optimistic end wouldn't change your quarter, the result is trivia regardless of its p-value.

That's the whole practice. No formulas, and it will keep you out of nearly every trap.

Why the tool should refuse to show you a winner early#

Given how strongly the peeking problem dominates, the useful design decision isn't better statistics — it's removing the opportunity. Bixlow declares the required sample when you configure and doesn't show a verdict before it arrives.

That's occasionally frustrating and it's the point. The alternative is a dashboard that lets a motivated team find a winner on day three, every time.

Key takeaways#

• Significance says how often chance alone would produce a gap this large. It is not the probability your variant is better.
• Peeking daily and stopping on green turns a 5% false-positive rate into 20–30%.
• Read the confidence interval, not the verdict. An interval containing zero is an inconclusive test.
• Statistically significant and commercially irrelevant are compatible. Always check the magnitude.