Mutually exclusive tests: stopping two experiments from contaminating each other

You're running a test on the product page and a test on the cart. A visitor can land in variant B of the first and variant A of the second, or B and B, or A and B, or A and A. You have four populations and you're analysing two.

Most of the time this is fine. Sometimes it quietly destroys your result, and the difference matters.

When overlap is harmless#

If two experiments touch unrelated parts of the journey and neither affects the other's metric, running them simultaneously costs you nothing and gets you two answers in the time of one.

A test on your blog's newsletter signup and a test on your checkout are independent. Randomisation distributes the blog variants evenly across both checkout variants, so the checkout comparison remains fair. This is the normal case and it's why parallel testing is standard practice at scale.

When overlap breaks the result#

It breaks when the experiments interact — when the effect of one depends on which variant of the other the visitor got.

The clearest case is two experiments in the same funnel with the same downstream metric. A product-page test that increases add-to-cart volume changes the composition of who reaches the cart. If your cart test is simultaneously changing the cart experience, the cart test is now being evaluated on a different population in each product-page variant.

Randomisation still balances the counts, so you won't get a badly biased average. What you lose is the ability to interpret it: the cart effect you measure is an average across two different upstream conditions, and if those conditions pull in opposite directions the average can be near zero while both real effects are large.

The second case is visual collision. Two experiments modifying the same region of the same page can produce a combination nobody designed and nobody previewed — a sticky bar from one test covering a promo banner from the other.

What mutual exclusion does#

Marking two experiments as mutually exclusive means a visitor enrolled in one is never enrolled in the other. Each keeps a clean, uncontaminated population.

The cost is traffic. The two experiments now split your audience rather than sharing it, so each gets half the sample and takes roughly twice as long. At a large company that's an easy trade. At a small brand it usually means you should have run them sequentially anyway.

In Bixlow you declare exclusivity between experiments and the assignment engine handles it deterministically — the same visitor always resolves the same way, with no race between two experiments trying to claim them.

Exclusion groups, not just pairs#

As a programme grows, pairwise exclusivity becomes unmanageable. The usual structure is an exclusion group: a named bucket where any visitor enters at most one experiment.

A sensible starting structure for an ecommerce brand is one group per funnel stage — discovery, product, cart, checkout — with experiments inside a group mutually exclusive and experiments across groups free to overlap. That captures most real interaction risk without paralysing your programme.

The honest advice for a small brand#

If you have low traffic, mutual exclusion is mostly a warning sign rather than a tool. Needing it means you're trying to run two experiments you don't have the sample for.

Run one at a time. It feels slower and it isn't: two half-powered simultaneous tests that both come back inconclusive have cost you more time than two sequential tests that both produced an answer.

Keep exclusion for the case where you genuinely must — a permanent personalisation that has to stay live while you test something else on the same page — and use it deliberately rather than as a default setting.

Key takeaways#

• Overlapping experiments are usually harmless. They break when one changes the population or the surface the other is measuring.
• Mutual exclusion buys you a clean population and costs you half your sample.
• Organise exclusivity as groups per funnel stage rather than as pairs.
• At low traffic, needing mutual exclusion usually means you should be running the tests sequentially.