The loop: how one experiment generates the next

Most experimentation programmes work like a queue. Ideas go in at one end, results come out of the other, and the tenth experiment is aimed no better than the first, because nothing from the first nine fed back into it.

A loop is different in one respect: each result changes what you test next. That doesn't happen on its own, and it's where most of the value builds up over time.

A win gives you a question about the mechanism. Say the sticky add-to-cart button won on mobile. Why did it win? If the CTA was simply out of reach, the same applies to your cart page and your lead forms, and you now have two well-founded hypotheses you didn't have yesterday. If people were using it to check the price, the next experiment is about price visibility, which is a completely different direction. A win you don't understand gives you nothing to follow up, which is why watching the recordings matters: it turns one improvement into a line of inquiry.

A loss gives you a constraint. You've learned something your customers don't want, and recorded properly that's worth as much as a win. "Adding the trust block above the fold lost 6%" tells you that on this page, space above the fold is worth more than reassurance, and three other ideas in your backlog just lost value.

A neutral result gives you a boundary. "No effect larger than 20% on add-to-cart" tells you the lever isn't there, so if you had two more ideas in the same area, they've just become less attractive.

A result that cancels out, positive for new mobile visitors and negative for returning desktop ones, gives you a targeting hypothesis. That's the most valuable outcome of the lot, since it points at a specific segment for a specific reason.

After every experiment, write one sentence about your customers rather than about the test.

"Variant B won by 8%" is a test result. "Our mobile buyers make up their minds before they reach the reviews" is something you know about your customers. The first is useful once; the second shapes every hypothesis you write for the next year.

Keep those sentences together in one list. Twelve experiments in, you have twelve statements about how your customers behave, and that document is worth more than the twelve results taken one by one. It's also what lets a new person on the team be useful in a week instead of a quarter.

The backlog you wrote in January rested on assumptions that January's experiments have since disproved. If you only review it once a quarter, it spends most of its life out of date.

After each result, go through the remaining candidates and ask which ones just became more likely or less likely. Usually two or three move, and now and then one drops off entirely because the result made it pointless. It takes ten minutes, and it's what keeps the programme a loop and stops it sliding back into a queue.

An agent can speed this part of the loop up, though it can't do it for you. It can notice that the segment that responded to your last experiment also behaves differently elsewhere on the site, and propose the follow-up test with that segment already declared.

What it can't do is decide what the win meant for your business. That sentence about your customers is a judgement call, and it's the input that makes the next proposal any good. An agent working from a well-kept list of customer learnings proposes much better experiments than one working from traffic data alone.

For a small brand running one experiment a month, the loop won't feel like it's building anything for the first six months. Four results don't add up to much.

Somewhere between eight and twelve results, that changes. You start to see patterns across experiments, your hypotheses get sharper and more of them win, because you're now testing things you have specific reasons to believe rather than because your luck improved. That's the real return on an experimentation programme, and you only get it if you've been writing things down since the first experiment.