Skip to content
Bixlow, AI CRO agent

Year one of experimentation, month by month

What the first twelve months of a small brand's CRO programme actually look like, including the stretch in the middle where it stops feeling worth the effort and most teams give up.

Experimentation programmes tend to get abandoned around month five, mostly because nobody warned the team what month five would feel like. This is a realistic outline of a first year for a site with roughly 300 to 800 sessions a day.

Months 1 and 2: setup and a first test

You get analytics working, instrument two or three custom events, install the experimentation script and record a baseline. Then comes the first experiment, ideally something structural on your busiest page.

It feels slow and administrative: two months gone and one test live. The usual mistake is to launch four tests at once out of impatience, which at this traffic gives you four inconclusive results and no answers at all.

Month 3: a first result that probably says nothing

Statistically, the most likely outcome of a first experiment is "no detectable difference", which surprises people who were expecting a verdict. After two months of setup and a month of waiting, a shrug is deflating.

Before you write it off, run the four checks. Were enough people exposed to it? Was the change visible? Did it render properly? Was it flat in every segment? Quite often the problem turns out to be exposure, in which case the honest conclusion is that the experiment needs running again, and the hypothesis may well be alive.

Months 4 and 5: the valley

By now you've run three experiments. One was inconclusive, one lost, and one won by too little to see in your revenue. Meanwhile a campaign pushed traffic up, so the numbers moved for reasons that have nothing to do with your tests. It feels like the programme isn't working, and this is where most of them stop.

It is working, though, and the payoff is weighted towards the end by its nature. Three results don't add up to a body of knowledge yet, and a single win disappears into the normal month-to-month swing in revenue; you can't see a 6% improvement in a month when traffic moved by 30%.

Keep going, and make sure someone is writing the learnings down as short statements about your customers. If nobody does, you'll reach month twelve holding twelve results and no accumulated understanding, and at that point it really won't have been worth it.

Months 6 to 8: it starts adding up

Somewhere around the sixth to eighth result, your hypotheses get noticeably better because you start recognising patterns. Mobile visitors decide earlier than you assumed. Paid traffic wants something different from organic. Your customers care more about delivery than about the material. For the first time it feels like you're building knowledge rather than placing bets.

This is the moment to use segment results on purpose. You now have enough history to know which segments behave differently, and experiments targeted at them become the most valuable ones you run.

Months 9 and 10: the first holdout

Now you find out whether any of it worked in aggregate. Hold back 5–10% of traffic from everything you've shipped, keep that control group for a month, and compare.

It's nerve-racking, as it should be. The likely result is less than the sum of your individual wins, which is normal: some wins were false positives, some novelty wore off, some effects overlapped. If the holdout shows a clearly positive effect, your process is sound. If it shows zero, you have a method problem, and you'd much rather discover it now than in year three.

Months 11 and 12: tidying up

Go through what's live. Ship winners as permanent changes and close their experiments. Switch off anything you can't justify with a metric, and write the year's learnings into a single document.

At this point you should have run ten to fifteen experiments, shipped three or four real wins, built a written set of statements about your customers and measured an aggregate effect. Don't expect a transformed conversion rate. At this scale, a cumulative 10–20% improvement in year one is a good outcome, and year two starts from a much better position because its experiments are aimed using what year one taught you.

What decides whether year two is better

Neither the tooling nor the win rate decides it. What matters is whether you wrote down what you learned.

A brand that starts year two with twelve sentences about how its customers behave will run much better experiments than one starting with twelve test results. The results go out of date within a season or two, while the sentences stay useful, so if you only keep one artefact from the year, keep those.

A twelve-month plan for your brand

Tell us your traffic and your goals and we'll map a realistic year: how many experiments, which ones first, and when to run your holdout. Free, no strings.

We'll only use your email to get in touch with you about this request.

AI CRO agent

Meet our AI CRO agent

What you just read, put to work by an agent: it understands your visitors, decides what to test or personalize, builds it on your site without a developer and tells you when a result is real.

Meet the agent