What is sample size in an A/B test
The sample size of an A/B test is the number of visitors each version needs to see so that the difference you measure is a real signal and not noise. It depends on your current conversion rate, how small a change you want to detect and how much certainty you ask of the result.
By Fernando Blanco Dosil · Updated November 2, 2026
Why you need a minimum sample
If you launch a test on Monday and by Tuesday the variant has 12 sales and the original 8, it looks like a win. But with so little data, a gap like that shows up by pure chance all the time. Flip a coin ten times and you will see seven heads without the coin being rigged.
Sample size answers a simple question: how many visitors do I need per version so that, if there really is an improvement of the size I care about, I will see it with reasonable certainty.
What drives it
There are four inputs, and it pays to know them before you launch anything.
- Baseline conversion rate: the lower it is, the more visitors you need. A store converting at 1% needs far more traffic than a lead-gen landing page at 10%.
- Effect you want to detect: detecting a 30% lift is cheap; detecting a 5% lift costs a huge amount of traffic. This is called the minimum detectable effect.
- Confidence: how often you accept being wrong when you call a winner. The usual figure is 95%.
- Power: the chance of seeing the lift if it exists. The usual figure is 80%.
A worked example with real numbers
Visitors per version at 95% confidence (two-sided) and 80% power, using the classic two-proportion formula.
| Current conversion | Relative lift you want to see | Variant conversion | Visitors per version |
|---|---|---|---|
| 2% | 10% | 2.2% | 80,683 |
| 2% | 20% | 2.4% | 21,109 |
| 3% | 10% | 3.3% | 53,211 |
| 3% | 20% | 3.6% | 13,914 |
Formula: n = (1.96·√(2p̄(1−p̄)) + 0.8416·√(p1(1−p1) + p2(1−p2)))² / (p2 − p1)², rounded up. With two versions, the total is double.
Common mistakes
The most common one is not calculating it at all and stopping the test when the chart looks good. The second is calculating it with an optimistic lift (30%, just because) and then being surprised the test never concludes: small changes, like button copy, rarely move that much.
Another classic: counting visits to the whole site when the test only touches the product page. The sample is the visitors who actually see the change, not your total traffic.
And don't forget full weeks. Even if you hit the sample in three days, a Tuesday visitor does not behave like a Sunday one, and a sale week is not a normal week.
How Bixlow handles it
The formula on this page is the classic estimate for planning a test. Bixlow judges results with Bayesian statistics: it only names a winner when each version reaches a minimum sample, the test runs for a minimum duration and the probability to be best reaches 95%. If something is missing, the results page says exactly what.