Someone checks the result every morning
The first day the variant is ahead, it gets crowned. A week later, the lead is gone.
Statistics and verdicts
Know when a result is real.
A test is only as good as its verdict. Bixlow tells you in plain language how likely each variant is to be best, how much you risk if you are wrong and whether there is enough evidence to decide.
Bixlow's statistics engine is a Bayesian engine that computes each variant's probability to be best, its expected loss and a 95% credible interval, and only declares a winner once minimum sample, minimum duration and a probability threshold are all met.
Conversion rate distribution
OriginalVariant B
- Probability to be best
- 96.4%
- Expected loss
- 0.02%
- 95% credible interval
- +2.1% … +9.8%
Verdict gates
- Minimum sample
- 7-day duration
- Probability ≥ 95%
- Approach
- Bayesian
- Gates before a verdict
- Three
- Probability required
- ≥ 95%
- Minimum duration
- 7 days by default
THE PROBLEM
How a good test goes wrong
It is rarely the idea that fails. It is the way the numbers get read.
Nobody knows what "significant" means
A p-value does not answer the question leadership is asking: how likely is B to be better, and what do we lose if it is not?
The split is broken and nobody notices
If one variant gets far more traffic than it should, something technical is wrong, and the result cannot be trusted.
Everyone runs their own math
One spreadsheet says one thing, the tool another and the report a third. The loudest voice in the room ends up deciding.
HOW IT WORKS
How Bixlow reaches a verdict
One engine reads every test with the same rules, and every result can be reproduced.
- 1
Count visitors and conversions
Every conversion is attributed to the variant the visitor saw, for the primary goal and for each guardrail.
- 2
Compute probabilities
A Beta-Binomial model for conversion rates and bootstrap for revenue. That gives the probability to be best, the expected loss and the credible interval.
- 3
Check the gates
Minimum sample per variant, minimum days, and at least 95% probability with expected loss under the threshold. While one is missing, there is no winner.
- 4
Get the verdict
Winner, no clear difference or waiting for data, with the reason in view. You decide whether to declare it and make it the base version.
CAPABILITIES
Statistical rigor in plain English
You do not need to know statistics to use it. You need someone to have done the statistics right for you.
Bayesian verdict
The answers you actually need
Instead of a p-value, you get numbers that make sense in a meeting: how likely it is to win and what is at stake.
- Probability that each variant is the best.
- Expected loss if you pick a variant that was not the best.
- A 95% credible interval for the effect.
Three gates
No premature winners
A good start is not enough. The engine waits for enough data and enough time before it speaks.
- Minimum sample per variant.
- Minimum duration, 7 days by default, to cover a full week.
- ≥95% probability with expected loss under the threshold.
Quality checks
Warnings before you make a bad call
The engine does not just look at who wins; it also checks that the test is healthy and that the winner does not break anything.
- Sample ratio mismatch (SRM) detection with chi-square, p < 0.001.
- Guardrails: a warning at 80% probability that a secondary metric is getting worse.
- Reproducible results: seeded Monte Carlo, same data, same verdict.
AI AGENT
The agent reads the verdict; it never makes one up
Neither the interface nor the AI computes its own verdict. Both report what the engine says, so what the chat tells you matches what you see on screen.
You decide
- Which goal is the primary metric.
- Which metrics must not get worse.
- When to declare the winner.
- What goes live and what gets dropped.
The agent does
- Summarizes the engine's verdict in plain language.
- Explains which gate is still open and why.
- Flags a guardrail or SRM warning.
- Proposes the next test based on the result.
A verdict in Bixlow
- Test
- Product page · Original vs. B
- Probability B is best
- 97%
- Expected loss choosing B
- Under the threshold
- Sample and duration
- Met · day 9
- Guardrail · revenue per session
- No warning
- Verdict
- B is the winner
BIXLOW STUDIO
Want someone to read the results with you?
Bixlow Studio designs tests so they can reach a verdict and reviews every result with a senior strategist before any decision is made.
Meet Bixlow Studio- 01Primary metric and guardrails chosen for each test.
- 02Verdicts read and a next step for every outcome.
- 03Monthly program report reviewed with a strategist.
FREQUENTLY ASKED QUESTIONS
Statistics
Why Bayesian instead of frequentist?
Because it answers the question you are actually asking: how likely a variant is to be best and how much you lose if you are wrong. The three gates also stop a winner being called on too little data.
What is expected loss?
It is how much the metric would drop, on average, if you pick a variant that was not really the best. A small expected loss means choosing it is a safe bet even when the difference is modest.
Why wait 7 days?
To cover a full week: behavior changes between weekdays and weekends. It is the default minimum duration, and it works alongside a minimum sample per variant.
What is an SRM?
A sample ratio mismatch: the real traffic each variant receives does not match the configured split. It usually points to a technical fault, and Bixlow warns you because that test's result is not reliable.
Can the AI tell me a variant won before the engine does?
No. The agent does not compute its own verdict: it reads the engine's and explains it. If the engine says the sample is not there yet, the agent says the same.
GET STARTED
Decide on evidence, not screenshots
Launch your first test and let the engine tell you when a result is real.
NO CARD · ONE SCRIPT · 10 MINUTES