Variant persistence: why a visitor must always see the same version

Persistence is the least glamorous part of an experimentation engine and the one that most often turns out to be broken. It's also invisible: a platform with inconsistent assignment produces results that look completely normal and mean nothing.

What has to hold#

Once a visitor is assigned to a variant, that assignment must survive: reloading the page, navigating to another page, closing the tab and returning tomorrow, and switching between pages of the same site.

If it doesn't, two things break at once.

The experience breaks. A customer sees one price presentation, comes back the next day, and the page is different. For anything with a consideration window longer than a single session — which is most purchases above impulse price — this is actively damaging. People notice, and it reads as untrustworthy.

The measurement breaks. Your conversion is attributed to whichever variant the visitor happened to be in at the moment they converted, not to the one that did the persuading. If someone saw B four times and A once, and converted on the A visit, variant A gets the credit for work variant B did. Across thousands of sessions this doesn't average out to something meaningful; it just adds noise weighted toward whichever variant people see on their final visit.

How assignment should work#

The right way is deterministic: take a stable visitor identifier, hash it together with the experiment identifier, and derive the variant from the hash. The same visitor and the same experiment always produce the same variant, without needing to look anything up.

This has a property that matters more than it sounds: there is no stored state that can be lost. You don't need a database lookup on every page view, and a visitor whose record is missing still resolves to the same variant they had before.

Hashing the experiment identifier together with the visitor is what keeps two simultaneous experiments independent. If you hashed only the visitor, the same people would always land in the "first" bucket of every experiment, and your experiments would be correlated in a way that quietly biases everything.

This is ordinary, strict, testable engineering, and it's why Bixlow's assignment lives in the deterministic engine rather than anywhere near the AI layer. The interface can be a language model; the arbiter cannot.

Where persistence actually fails#

Cookie clearing and private browsing. Some loss is unavoidable. It's usually small and roughly equal across variants, so it adds noise rather than bias.

Cross-device. Someone browsing on a phone and buying on a laptop is a different visitor to your engine. Unless you can join identities after login, they may see both variants. This is a real limitation, not a bug, and it's worth knowing your cross-device rate before over-interpreting small effects.

Changing the traffic split mid-experiment. Moving from 50/50 to 80/20 while running will reassign some visitors on most naive implementations. Deterministic hashing with stable bucket boundaries avoids the worst of it, but the honest answer is: don't change the split on a running experiment.

Adding a variant mid-flight. This reshuffles everyone. Never do it — stop and start a new experiment.

What to check in any tool#

Three questions worth asking before you trust a platform's numbers:

Does assignment survive a full browser restart? Open the site, note the variant, quit the browser, come back.

Is the same visitor consistent across pages? Navigate the whole funnel and confirm the variant doesn't change between the product page and the cart.

Does changing the traffic allocation reassign existing visitors? If the tool lets you change it at all on a live test, ask what happens to people already enrolled.

Key takeaways#

• A visitor must keep their variant across reloads, pages, sessions and days, or both the experience and the measurement break.
• Deterministic hashing of visitor plus experiment removes the need for stored state and keeps parallel experiments independent.
• Cross-device switching is a real limitation, not a bug. Know your rate before over-reading small effects.
• Never change the traffic split or add a variant on a running experiment.