From zero to a hundred experiments: how a programme actually scales

The simple picture of growing an experimentation programme is that you run more tests. It's the wrong picture, because what holds you back changes as you grow, and each stage breaks for its own reason.

In the early stage the limit is traffic. You can measure one thing at a time, and each test takes three to six weeks.

So choosing well is most of the job, because every test is about a tenth of what you'll learn that year. Everything we've written about prioritisation applies here, and the commonest mistake is spending those slots on cosmetic tests whose effect could never have been detected.

It's tempting to get round the traffic problem by running several tests in parallel. At low traffic, that's how you end up with four inconclusive results where you could have had two answers.

Somewhere around here, traffic can carry two or three experiments at once, and a new problem appears: they start interacting.

This is when exclusion groups earn their keep. Organise them by funnel stage (discovery, product page, cart, checkout), make experiments within a group mutually exclusive, and let experiments in different groups overlap. That covers most of the real interaction risk without slowing you to a crawl.

Something else breaks at this stage that nobody sees coming: people forget what's live. You need one list of running experiments, each with an owner, a hypothesis, a metric and an end date. A spreadsheet is perfectly good enough, but you do need one.

Beyond roughly forty experiments, traffic stops being the main limit. Understanding takes its place.

By now twenty winners have been shipped and are changing the site permanently. Some of them were targeted at segments, so different visitors see different things. Half of them were launched before anyone on the current team arrived. When something behaves oddly, it takes a day to work out why.

Programmes tend to die quietly here. People stop testing in certain areas because they don't understand what's already there, new hires take a quarter to become useful, and results get stranger because each test sits on top of a foundation nobody can describe any more.

What fixes it is dull work. Keep a register of live modifications: what changed, for whom, which experiment justified it and which metric it moved. Review it every quarter and switch off anything you can't justify.

Ship winners as code instead of leaving them as permanent experiments. An experiment left at 100% is a modification hiding inside your testing tool; once something wins, it should become part of the site and the experiment should be closed.

And keep a learnings document separate from the results, made of the sentences you now know about your customers rather than the outcome of each test. That document is what gets a new person up to speed fast.

People tend to overinvest in process. A programme running four tests a month doesn't need an intake form, a scoring committee and a weekly review meeting. It needs a prioritised list and one person who owns it.

They also overinvest in tooling. Most of the advanced features in enterprise platforms exist for organisations where dozens of teams touch the same site. A single team needs targeting, assignment, statistics and a way to build variants without a developer, and the rest is built for a company you aren't.

For a brand with modest traffic, ten to fifteen well-run experiments a year makes a good programme, and that's simply what your sample can support. The count isn't what builds up over time. What builds up is whether your fifteenth experiment is better aimed than your first, and that depends far more on the learnings document than on how many tests you get through, so start that document with the next result you get.