# What is sample ratio mismatch (SRM)

> Sample ratio mismatch (SRM) is what happens when an A/B test does not split visitors the way it should: you set 50/50 and one version gets noticeably more visitors than the other. It is the clearest sign that a test is broken, and while it lasts, the result cannot be trusted.

- Source: https://bixlow.com/en/glossary/sample-ratio-mismatch
- Language: en
- Updated: 2026-11-02
- Publisher: Bixlow

## Why it matters so much

A 50/50 split is never exact: 10,000 against 9,900 is normal. But when the gap is too large to be chance, something is steering a particular kind of visitor away from one of the versions.

And that something is rarely neutral. If the variant loses visitors on slow connections, or those coming from a specific campaign, the two groups are no longer comparable. The variant can look like a winner only because the worst-converting visitors were taken out.


## A worked example

A test set to 50/50. The chi-square test says whether the gap could be chance.

| Visitors in the original | Visitors in the variant | Test p-value | What it means |
| --- | --- | --- | --- |
| 10,000 | 9,900 | 0.48 | Normal split, chance explains it |
| 10,000 | 9,500 | 0.0003 | SRM: the test has a problem |

_With 19,500 visitors you would expect 9,750 on each side. A deviation of 250 per side at that volume happens by chance about 3 times in 10,000._


## The usual causes

It is almost always a technical or tracking problem, not a statistical one. These are the usual suspects.

- Bots: a crawler that only runs one of the versions, or that keeps coming back with different identities, inflates one side.
- Redirects: if the variant sends people to another page, some of them get lost on the way and are never counted.
- Tracking failures: the variant breaks the analytics script, or loads so late that the visitor leaves before being recorded.
- Caching: a version served from cache to someone who should not get it, or a cached page that never runs the test.
- Changing the split mid-test: moving from 50/50 to 80/20 and mixing both phases in the same result.


> **What to do when it fires.** Don't look at who is winning. Find the cause: compare the split by device, browser and traffic source, and check whether the imbalance starts on a specific day (a deploy, a new campaign). Once it is fixed, restart the measurement from zero. The data you already have is contaminated.


## How Bixlow handles it

Bixlow runs a chi-square test of exposures against the split you configured, with a 0.001 threshold to avoid false alarms. If it fails, the results page says so and gives no verdict, even if one version seems to be winning. And if you change the split of a running test, measurement starts again so the two phases are never mixed.


## FAQ

### Does an SRM favour one of the versions?

Not in itself. It tells you the groups are no longer comparable, so the result may be biased in either direction.

### How much imbalance is normal?

It depends on volume. With a few hundred visitors, 55/45 can be chance; with tens of thousands, 51/49 can already be a problem. That is why you use a statistical test and not your eyes.

### Can I correct the result instead of rerunning the test?

It is not a good idea. You don't know which visitors are missing or how they would have converted. Fix the cause and measure again.

### Why a 0.001 threshold and not 0.05?

Because the check is repeated many times during the test. At 0.05, one test in twenty would raise the alarm for no reason.

## Further reading

- [What is Bayesian A/B testing](https://bixlow.com/en/glossary/bayesian-ab-testing)
- [Bixlow's statistics](https://bixlow.com/en/platform/statistics)
- [A/B testing with Bixlow](https://bixlow.com/en/platform/ab-testing)