What to Do When a Test Returns Inconclusive Results

Not every test produces a clear winner. Often, after weeks of experimenting, the two versions end up practically tied and the tool refuses to declare significance. It's a frustrating situation, but it isn't a failure: an inconclusive result is information too, and knowing what to do with it is what separates teams that learn from teams that merely test.

The key is not to react rashly, neither by forcing a winner that doesn't exist nor by throwing away the learning. In this article you'll see how to correctly interpret a flat test, what questions to ask yourself to understand what happened, and which paths lie ahead depending on the reason for the tie.

An inconclusive result can mean two very different things. It may be that there's no real difference between the versions, meaning your change doesn't affect user behavior. Or there may be a difference, but one so small, or measured with so little sample, that the test lacked the power to detect it. Telling these two cases apart is the first step in deciding what to do.

To orient yourself, look at the sample size you reached and the confidence interval. If you gathered the planned sample and the interval is narrow around zero, there probably really is no effect. If the sample fell short or the interval is very wide, the problem is one of power and the test was in no position to conclude anything.

Once you understand why there's no winner, you have several reasonable options. The decision depends on how much confidence you need and how much the change matters to your business.

• If you lacked sample, extend the test or relaunch it on a higher-traffic page.
• If the change was small, try a bolder variant that produces a larger effect.
• If there truly is no difference and the change is neutral, pick the version that's simplest or cheapest to maintain.
• If the result contradicts your hypothesis, revisit that hypothesis before iterating further in the same direction.

A test with no winner disproves a belief, and disproving beliefs has value: it saves you from investing in changes that don't move the needle. Document what you tested, what you expected, and what happened, so the team doesn't repeat the experiment six months from now. A well-documented tie sharpens your intuition about what really matters to your users and improves the quality of your next hypotheses.

• A tie can come from there being no real effect, or from a lack of power and sample.
• Look at the sample you reached and the confidence interval to tell the two cases apart.
• Document the flat result: disproving a belief is valuable learning too.