# The most common reason a test comes back flat: nobody noticed > Before you decide a flat test means the idea was wrong, check whether anyone saw it. A lot of null results come from the change never reaching people, and the hypothesis was never really tested. - Source: https://bixlow.com/en/blog/nobody-noticed-your-change - Language: en - Published: 2026-10-27T09:00:00.000Z - Updated: 2026-10-06T09:28:18.400Z - Author: fernando - Topics: experiment design, session recordings, A/B testing, minimum detectable effect, CRO, Microsoft Clarity, audience targeting - Publisher: Bixlow --- A null result feels like an answer: we tested it, it didn't work, next. It's a tidy conclusion and it's wrong more often than people think. The idea may have been no good, but quite often the experiment simply never put it in front of enough people to find out. Before you file a flat test as a rejected hypothesis, rule out four things. ## The change was below the fold Look at scroll depth at the position of your change, device by device. If 30% of mobile visitors get that far, the other 70% of your mobile sample saw no difference at all between the two variants. The arithmetic is worth keeping in your head. With only a third of visitors exposed, a real 15% effect among those who saw it shows up as roughly 5% in the headline figure, and 5% is probably below what the test was sized to detect. The effect was there; it got diluted until it disappeared. You can move the change above the fold and run the test again, or restrict the experiment to visitors who actually reached that point so that the sample is the exposed population. The second option is cleaner but needs the scroll event instrumented. ## The change was too small to register People scan web pages more than they read them. A rewritten paragraph in the middle of a block of text, for most visitors, didn't change anything. This is the exposure side of the usual advice to test structure rather than decoration. If screenshots of the variant and the control look nearly the same at a glance, expect a null result however much better the new copy is. Put the two side by side and look for two seconds. If you can't tell them apart in that time, your customers can't either. ## The change didn't render everywhere This one is a bug rather than a design problem, and it's more common than anyone expects. A variant that fails to apply on Safari, at a certain viewport width, or whenever the cookie banner is showing leaves you with a mixed group where some of your "variant" visitors actually saw control. That drags the measured effect towards zero mechanically, and the results panel won't show it, because those sessions still count as variant. A health check partway through the test catches it. Watch a handful of recordings from the variant group in the first two days and make sure the change is really there. It takes five minutes, against three weeks of measuring nothing. ## It mattered to a segment We've written about this at length elsewhere, but it belongs on the list. Read the breakdown before concluding anything. A change that only matters to first-time mobile visitors will look flat across an audience that's mostly returning desktop users. ## When the null is real Once you've ruled out those four, a flat result is genuine and worth having. It says that, within what you were able to detect, this change doesn't matter to your customers. Record it properly. "No effect larger than 20% on add-to-cart, exposure confirmed for 85% of the sample" is a useful record. "Didn't work" is a note you'll misread in six months, and maybe end up testing the same thing again by accident. Real nulls are also how you stop wasting effort. A team that knows three specific things don't move its customers knows more than one that has only ever written down wins. ## A five-minute check The cheapest way to do all this is a short routine at the end of every experiment, before you interpret anything. What share of the sample was actually exposed to the change? Did it render on every device and browser in the sample? Is the effect flat in every segment, or only flat on average? Would you have noticed the change yourself as a visitor? Asked consistently, those questions turn a good number of "the idea was wrong" verdicts into "the experiment needs running again, properly", and the second one leaves you with something to do next. --- ## Meet our AI CRO agent What you just read, put to work by an agent: it understands your visitors, decides what to test or personalize, builds it on your site without a developer and tells you when a result is real. https://bixlow.com/en/ai-cro-agent