Mistakes that invalidate your A/B tests (and how to avoid them)

A poorly run A/B test is worse than running no test at all, because it gives you a false sense of security and pushes you to make wrong decisions with the confidence of someone who believes they have data. Many mistakes are invisible at first glance: the experiment looks correct, returns a winner, and only later do you discover the result wasn't real.

Most of these failures are avoidable if you know about them in advance. In this article we go through the most common mistakes that invalidate results and how to prevent them, so that every experiment you launch becomes a solid foundation to build on rather than a source of misleading conclusions.

Design and configuration mistakes#

A good share of the problems are born before launch, in the design phase. If the experiment starts from a flawed foundation, no later analysis will save it. Review these points before you hit publish.

• Changing several elements at once, leaving you unable to attribute the result to any single one.
• Not defining the primary metric before starting and instead picking it afterward based on whatever comes out as the winner.
• Launching without calculating the sample size or the required duration.
• Splitting traffic in a non-random way, biasing who sees each version.

Mistakes during execution#

Other failures show up while the test is live. The most damaging is stopping it the moment one version seems to be winning, a practice known as peeking that triggers false positives. It's also common to contaminate the experiment by making changes to the site during the test, launching campaigns that alter the type of traffic, or mixing users who see both versions due to poor handling of cookies or sessions.

To avoid this, set the duration and sample size in advance, don't touch anything in the test while it runs, and make sure each user always sees the same version. If you need to change something, the right move is to stop the experiment, document it, and relaunch it clean rather than altering one that's already running.

Interpretation mistakes#

Even when the test is well executed, you can still misread it. Confusing statistical significance with business relevance, ignoring that a global result can hide opposite behaviors across segments, or generalizing the conclusions of a one-off test to your entire strategy are frequent traps. Always analyze with a critical eye and, when faced with a surprising result, validate before acting.

Key takeaways#

• Define your metric, sample size, and duration before launching, and don't change them afterward.
• Don't stop the test early or make changes to the site while it's running.
• Interpret carefully: significance isn't relevance, and a global result can hide segment-level differences.