The interface can be AI. The engine underneath cannot.

It's tempting to assume that if AI is good at a task, more AI makes it better. Apply that to experimentation and you get a system you can't trust, and the reason has to do with how the system is built rather than with how good the model is.

Split the system into two layers.

The upper layer is where AI fits. Understanding what you asked for in plain language, generating the workflow that makes a change, reading traffic to find segments that diverge, proposing a hypothesis and a mechanism, summarising what a result means. The output of each of these is a proposal, and a person reviews it before it has any effect. If the model gets something wrong, you see it in the blueprint and fix it.

The lower layer is where AI should never go. Which visitor gets which variant; whether someone in experiment A can also enter experiment B; how many sessions make a valid sample; whether a difference is significant; whether a guardrail has degraded; how an event gets counted. These produce facts that decisions rest on, and nobody reviews them, because reviewing them takes exactly the expertise you bought the tool so you wouldn't need.

The lower layer needs a few properties that no language model has.

The first is reproducibility. The same visitor has to land in the same variant every single time. A deterministic hash guarantees that by construction, while a model doesn't guarantee it at all, and once assignment can vary, persistence is broken.

The second is auditability. "Why is this significant?" needs an answer you can follow and check. A fixed formula with declared thresholds gives you one. A model's judgement doesn't, and "the model said so" isn't something you can defend to anybody.

The third gets discussed least and matters most: the statistics can't be open to persuasion. You want your winner to be real. A model in a conversation with a motivated user drifts towards agreeing, which is what makes it pleasant to work with and also what rules it out as a referee. The statistics have to be the one part of the system that nobody can talk into anything.

This runs against intuition. The better the AI layer gets, the stricter the engine underneath has to be.

When a person configures every experiment by hand, they act as a slow, expensive brake on bad ideas. When an agent can propose twenty experiments a week, that brake has gone, and the only thing between a plausible-sounding proposal and a wrong business decision is whether the measurement underneath is honest. So the deterministic, dull, heavily tested infrastructure is what makes the AI safe to use, however unglamorous it looks next to the AI part of the product.

Assignment in Bixlow is a hash of the visitor and the experiment that produces a bucket. It's a handful of lines with unit tests, and it does the same thing every time. There's nothing clever about it.

Exclusion groups are checks on set membership. The verdict comes from a seeded Bayesian model with explicit gates: a minimum sample per variant, a minimum duration, the probability of being best and the expected loss, plus a check that the traffic split hasn't drifted. Guardrails are comparisons against a threshold. None of it is interesting engineering and all of it can be tested. When a result says a variant won, the claim rests on code you could read in an afternoon rather than on a model's opinion.

If you're looking at any experimentation tool with AI on the label, a few questions will tell you which kind of system it is. Is visitor assignment deterministic, and can they describe how it works? How is significance calculated, and what does a result have to pass before they call it a winner? Can the AI layer change a result, a threshold or an assignment? Where exactly does what the model proposes end and what the engine decides begin?

If a vendor can't show you clearly where that line sits, assume they never drew one.