Segment results: the neutral test that isn't neutral

A flat overall result is the most common outcome of an experiment and the least examined. The team records "no effect", moves on, and throws away the most interesting information the test produced.

Because a flat average has two very different explanations. Either nothing happened anywhere — or something happened in opposite directions and the average erased it.

Genuinely nothing. Every segment sits near zero, all intervals overlap zero, the change didn't matter to anybody. This is a real finding. Record it and move on.

Cancellation. New mobile visitors are +14%, returning desktop visitors are −9%, and the weighted average is +0.6%. The change helped one group and hurt another. Filing this as "no effect" means you throw away a win and keep a loss.

You can only tell these apart by looking at the breakdown. Bixlow splits every result by device, visitor type, country and traffic source whether or not you targeted them, specifically so that the second case can't hide inside the first.

Here's the tension. Looking at segments after the fact is also the single easiest way to fool yourself — with a dozen slices, one will look significant by chance even when nothing is happening.

So the breakdown has two legitimate uses and one illegitimate one.

Legitimate: diagnosis. A large negative in one segment tells you the change has a cost you didn't anticipate. You don't need statistical certainty to take that seriously — it's a reason to look at the page on that device, not a reason to ship anything.

Legitimate: hypothesis generation. "New mobile visitors were +14%" becomes the next experiment, declared up front, targeted at exactly that segment. If it survives a proper test, it's real.

Illegitimate: shipping it. Taking the winning slice and deploying to it because the number looked good is the multiple-comparisons trap, and it's how teams accumulate personalisations that were never actually validated.

When you do find cancellation, the mechanism is usually one of these:

The change assumes context one group doesn't have. A test that leans on prior familiarity helps returning visitors and confuses new ones.

The change costs space. Adding a trust block above the fold helps visitors who needed reassurance and pushes the product further down for visitors who were already sold. On mobile, where everything is a sequence, this is very common.

The change matches one intent. Simplifying toward a single call to action helps people who came to buy and frustrates people who came to compare.

Notice that all three are differences in what the visitor needed, which is exactly what audience targeting exists to address. A cancelling result isn't a failed experiment; it's an argument for segmentation that you didn't have before.

Three steps, in order.

First, write down the segmented result even though you're not acting on it. "Neutral overall; +14% new mobile, −9% returning desktop, both wide intervals" is a note worth having.

Second, decide whether the negative segment is a reason not to ship at all. If the change is neutral overall and clearly harms a group you care about, neutral is not good enough.

Third, if the positive segment is plausible and large, run it as a targeted experiment. Declare the segment, declare the sample, and find out. That's a month well spent — you're chasing an effect you have a specific reason to believe in, which is a better starting point than most hypotheses ever get.

• A flat average has two explanations: nothing happened, or opposite effects cancelled. Always read the breakdown.
• Segment results are for diagnosis and for generating the next hypothesis. They are not a licence to ship the winning slice.
• Cancellation usually means different segments needed different things — which is an argument for targeting.
• Re-run the promising segment as a properly declared targeted test. If it survives, it's real.