Sample size: how to know if your test is reliable

Sample size is the factor that ruins the most experiments without anyone noticing. A test can be perfectly designed, with a flawless hypothesis and a well-executed variant, and still be worthless if it doesn't gather enough users for the result to be reliable. Understanding this concept saves you weeks of wasted work.

The good news is you don't need to solve any formulas: you just need to know three variables and use a calculator. The bad news is that many teams launch tests without checking whether they have the traffic to finish them, and end up making decisions on data that doesn't hold up to scrutiny. In this article you'll see how to avoid that.

The three variables that determine sample size#

The sample size you need depends on three things. The first is your current conversion rate, or baseline rate: the lower it is, the more users you need. The second is the minimum detectable effect, meaning the smallest improvement you want to be able to detect; spotting a 1% change requires far more sample than spotting a 10% one. The third is the confidence level, usually 95%.

With those three inputs, any sample size calculator returns the number of users per variant you need. Running this calculation before launching is what turns a test into a sensible investment, because it tells you up front whether you have enough traffic to reach a conclusion within a reasonable timeframe.

What happens when you have low traffic#

If your site gets little traffic, tests with small effects can take months or be outright unfeasible. Instead of giving up, adjust your strategy so your experiments keep generating value despite the volume limitation.

• Test big, bold changes, which produce larger effects and need less sample.
• Focus the test on the pages or steps with the most traffic, not on rarely visited corners.
• Measure more frequent metrics, like clicks, instead of scarce final conversions.
• Accept a slightly lower confidence level for reversible, low-risk decisions.

Don't confuse sample with time#

Reaching the sample size isn't enough on its own: that volume has to accumulate across complete cycles to be representative. Gathering 30,000 users in two days of a campaign isn't the same as gathering them over two normal weeks. Always combine the sample criterion with the duration criterion to make sure the data reflects your real audience and not a one-off situation.

Takeaways#

• Sample size depends on the baseline rate, the minimum detectable effect and the confidence level.
• Calculate the sample you need before launching so you know whether the test is viable.
• With low traffic, test big changes and focus on your most visited pages.