# The control group and the holdout: proving your work did anything

> Individual tests tell you whether one change worked. A holdout tells you whether a year of changes did — and the answer is sometimes uncomfortable.

- Source: https://bixlow.com/en/blog/control-group-holdout-what-you-did
- Language: en
- Published: 2026-10-17T09:00:00.000Z
- Updated: 2026-10-06T09:28:18.400Z
- Author: fernando
- Topics: control group, experiment design, experimentation program, A/B testing, statistical significance, CRO, mutual exclusion
- Publisher: Bixlow

---
Your control group is the version of reality where you did nothing. Everything you claim about your experiment is a comparison against it, which makes it the most important and least discussed part of the setup.

## Why "before and after" isn't a control

The most common substitute for a control group is comparing this month to last month. It is almost always wrong, because between the two months your traffic mix changed, your ad spend changed, the season changed, and possibly a competitor did something.

A proper control is *concurrent*: the same days, the same campaigns, the same weather, the same everything — differing only in which version of the page they saw. That simultaneity is the whole point. Anything measured sequentially is contaminated by time.

This matters most for the results that feel most convincing. "We redesigned the checkout and conversion went up 20%" is a sentence with no control group in it, and it's the kind of claim that shapes a year of roadmap.

## The accumulation problem

Here's what individual experiments can't tell you. Over a year you run twelve tests, seven win, and each win is 3–8%. Multiply those together and you should be up 30% or more. Are you?

Usually not, and there are honest reasons. Some "wins" were false positives. Some effects decayed as novelty wore off. Some changes interacted and their gains overlapped rather than stacked. And your baseline drifted for reasons unrelated to any of it.

The only way to find out is a holdout: a small slice of traffic — typically 5–10% — that is systematically excluded from everything you ship, for a long period. At the end of the quarter you compare the holdout against everyone else, and that difference is your programme's actual cumulative effect.

## What a holdout costs and what it's worth

It costs you the improvement those visitors didn't get. If your changes really are worth 20%, you're leaving that on the table for 5% of traffic. That's a real, quantifiable cost.

What you buy is the only honest answer to "is any of this working?". For a small brand deciding whether to keep investing time in experimentation, that answer is worth having, and it's also the number to show anyone asking whether CRO is worth it.

The practical compromise: run a holdout for a quarter, not permanently. Two or three measurement points a year is enough to know whether your programme compounds, without permanently taxing your revenue.

In Bixlow a holdout is an audience permanently excluded from a set of experiments — the same exclusion machinery as mutually exclusive tests, pointed at a persistent group instead of a temporary one. Persistence matters enormously here: a holdout visitor who drifts into a variant contaminates a measurement you'll only take once a quarter.

## When the answer is disappointing

Sometimes the holdout shows less lift than your individual tests promised. This is not a reason to stop experimenting; it's a reason to trust your process more and your individual results less.

Concretely, it usually means: raise your confidence threshold, stop shipping marginal wins, run longer tests on fewer hypotheses, and check whether your "wins" were concentrated in one segment and diluted when deployed everywhere.

A programme that knows its real cumulative effect is in far better shape than one that adds up twelve optimistic percentages and believes the total.

## Key takeaways
- A control group must be concurrent. Comparing this month to last month is not a control.
- Individual wins don't multiply cleanly: false positives, novelty decay and overlapping effects all eat the total.
- A 5–10% holdout for one quarter is the only honest measure of what your whole programme achieved.
- A disappointing holdout means tighten your process, not abandon it.

---

## Meet our AI CRO agent

What you just read, put to work by an agent: it understands your visitors, decides what to test or personalize, builds it on your site without a developer and tells you when a result is real.

https://bixlow.com/en/ai-cro-agent