What is sample ratio mismatch (SRM)
Sample ratio mismatch (SRM) is what happens when an A/B test does not split visitors the way it should: you set 50/50 and one version gets noticeably more visitors than the other. It is the clearest sign that a test is broken, and while it lasts, the result cannot be trusted.
By Fernando Blanco Dosil · Updated November 2, 2026
Why it matters so much
A 50/50 split is never exact: 10,000 against 9,900 is normal. But when the gap is too large to be chance, something is steering a particular kind of visitor away from one of the versions.
And that something is rarely neutral. If the variant loses visitors on slow connections, or those coming from a specific campaign, the two groups are no longer comparable. The variant can look like a winner only because the worst-converting visitors were taken out.
A worked example
A test set to 50/50. The chi-square test says whether the gap could be chance.
| Visitors in the original | Visitors in the variant | Test p-value | What it means |
|---|---|---|---|
| 10,000 | 9,900 | 0.48 | Normal split, chance explains it |
| 10,000 | 9,500 | 0.0003 | SRM: the test has a problem |
With 19,500 visitors you would expect 9,750 on each side. A deviation of 250 per side at that volume happens by chance about 3 times in 10,000.
The usual causes
It is almost always a technical or tracking problem, not a statistical one. These are the usual suspects.
- Bots: a crawler that only runs one of the versions, or that keeps coming back with different identities, inflates one side.
- Redirects: if the variant sends people to another page, some of them get lost on the way and are never counted.
- Tracking failures: the variant breaks the analytics script, or loads so late that the visitor leaves before being recorded.
- Caching: a version served from cache to someone who should not get it, or a cached page that never runs the test.
- Changing the split mid-test: moving from 50/50 to 80/20 and mixing both phases in the same result.
How Bixlow handles it
Bixlow runs a chi-square test of exposures against the split you configured, with a 0.001 threshold to avoid false alarms. If it fails, the results page says so and gives no verdict, even if one version seems to be winning. And if you change the split of a running test, measurement starts again so the two phases are never mixed.