What is Bayesian A/B testing
Bayesian A/B testing means analysing a test with Bayesian statistics: instead of a p-value, it tells you directly how likely each version is to be the best, how much it probably improves and how much you would lose if you picked the wrong one.
By Fernando Blanco Dosil · Updated November 2, 2026
The idea, in two sentences
The Bayesian method starts from what you know before the test (often, almost nothing) and updates it with every visitor and every conversion. At the end you have a distribution of plausible conversion rates for each version, and with it you can answer business questions.
The question everyone asks after a test is: which one wins, and how sure are we? The Bayesian approach answers exactly that. The classic one answers something else, and it has to be translated.
Bayesian versus frequentist
| Frequentist (classic) | Bayesian | |
|---|---|---|
| Main result | p-value and significance level | Probability that each version is the best |
| How you read it | It needs translating, and is often misread | Read it as is: 97% means a 97% chance it wins |
| How much it improves | Confidence interval | Credible interval: where the real lift is with 95% probability |
| Risk of being wrong | Not quantified | Expected loss: how much conversion you lose if you pick wrong |
| Checking results often | Inflates false winners | Tolerated better, always with minimum sample and duration |
The three numbers you will see
A good Bayesian report does not stop at one number. It gives you three, and each answers a different doubt.
- Probability to be best: of all versions, which one is most likely to convert more. If the variant has 96%, it wins in 96 out of every 100 scenarios consistent with the data.
- Expected loss: if you pick this version and it turns out not to be the best, how much conversion you lose on average. When it is close to zero, picking it is a safe bet even if the probability is not sky-high.
- Credible interval: the range where the real lift probably sits. A +8% lift with an interval from +2% to +14% is solid; from −5% to +21%, not yet.
What it does not fix
Bayesian does not mean you can decide with 300 visitors. With a small sample, the probability swings a lot from one day to the next. That is why a good Bayesian engine also requires a minimum sample per version and a minimum duration, so the test covers at least a week with its good and bad days.
Nor does it save a badly set-up test. If a variant fails to load for some visitors, no calculation fixes it. That is what the sample ratio mismatch check is for.
How Bixlow handles it
Bixlow analyses every test with a Beta-Binomial model and a uniform Beta(1,1) prior, meaning it assumes nothing about your conversion rate before the test starts. It only declares a winner when the probability to be best reaches 95% with a low expected loss, each version reaches the minimum sample and the test meets the minimum duration. If the sample ratio mismatch check fails, it gives no verdict.