# Why most "winning" tests don't survive contact with reality > Four mechanisms turn a real-looking result into a lift you never see in your revenue: peeking, novelty, regression to the mean and segment cherry-picking. - Source: https://bixlow.com/en/blog/why-most-shipped-winners-dont-replicate - Language: en - Published: 2026-10-18T09:00:00.000Z - Updated: 2026-10-06T09:28:18.400Z - Author: fernando - Topics: statistical significance, experiment design, A/B testing, control group, CRO, experimentation program, audience targeting - Publisher: Bixlow --- Large experimentation programmes that have measured this honestly tend to report the same uncomfortable thing: a large share of shipped winners don't reproduce when re-tested. The gap between what experiments promise and what the business shows up is not a mystery. It has four specific causes, and three of them are avoidable. ## 1. You stopped when it looked good This is the dominant cause and it's worth stating again in a different way. A 95% confidence threshold assumes one look, at a pre-declared sample. Checking daily and stopping on green turns that into a procedure with a real false-positive rate around 20–30%. So if you peeked, roughly one in four of your "winners" was never a winner. You didn't measure an effect; you measured the most flattering moment of a random walk. The fix is procedural and absolute: declare the sample, don't look, don't stop early except to prevent harm. Bixlow withholds the verdict until the declared sample arrives for precisely this reason — the discipline fails when the temptation is one click away. ## 2. The novelty effect Returning visitors notice change. A new banner, a restructured page, a different button placement all attract attention simply by being different from what someone saw last week. Attention looks like engagement, and engagement looks like a win. Then it fades. Two months later the variant is just the page, and the lift is gone. This mostly affects sites with a high proportion of returning visitors. Two ways to detect it: check whether the effect is concentrated in returning visitors — if new visitors show nothing and returning visitors show a large lift, suspect novelty rather than improvement. And look at whether the effect decays across the weeks of the test rather than staying flat. Bixlow splits results by visitor type by default, which makes the first check free. ## 3. Regression to the mean You picked the page to test because it was underperforming. Underperformance is partly real and partly bad luck in the period you measured. When you test it, the bad luck doesn't repeat, and that portion of the recovery gets attributed to your change. This is why "we fixed our worst page and it improved" is weaker evidence than it feels. The page would have improved somewhat anyway. A concurrent control group handles this correctly — which is exactly what an A/B test is, and exactly what a before-and-after comparison is not. If your evidence is sequential rather than concurrent, assume some of the improvement was never yours. ## 4. You found the segment afterwards The test is neutral overall. You slice by device, country, traffic source, new versus returning and price band. One slice shows +22%. You ship it for that slice. With a dozen slices, finding one that looks significant at 95% is close to guaranteed even when nothing is happening. This is the multiple-comparisons problem, and post-hoc segmentation is its most seductive form because the story always sounds plausible afterwards. The rule that makes segment analysis useful rather than misleading: a striking sub-group result is a **hypothesis for the next experiment**, never a decision. Re-run it as a targeted test with that segment declared up front. If it survives, it's real, and now you know something valuable. If it doesn't, you just saved yourself from shipping noise. ## What to do about it The practical consequence for a small brand is to ship fewer, more certain things. Raise your bar: require the confidence interval to exclude zero comfortably, not marginally. Prefer one clear result a quarter to four ambiguous ones. And run a holdout once or twice a year. It's the only way to find out whether your programme's claimed gains show up in aggregate — and if they don't, the four mechanisms above are where to look. ## Key takeaways - If you peeked, roughly a quarter of your winners were noise. The declared-sample rule is not optional. - Novelty inflates results on sites with many returning visitors and fades. Check whether the effect is concentrated there. - Testing your worst page overstates your contribution unless the control is concurrent. - A striking sub-group result found after the fact is a hypothesis, not a decision. Re-run it targeted. --- ## Meet our AI CRO agent What you just read, put to work by an agent: it understands your visitors, decides what to test or personalize, builds it on your site without a developer and tells you when a result is real. https://bixlow.com/en/ai-cro-agent