What a CRO programme really costs a small brand

Most comparisons of experimentation tools line up monthly prices, which is about the least interesting number in the whole decision. Below is the full set of costs for a small brand that takes testing seriously, and what a tool has to do to earn its place.

Hours come first, by a distance. Someone has to write hypotheses, check what's about to go live, read results and, the step everyone skips, write down what was learned. For a small programme that's realistically four to eight hours a month. It isn't a full-time job, but it isn't nothing either, and unless it belongs to one named person it won't get done.

Then there's the traffic spent on tests that don't conclude. Every experiment that ends without an answer used up sample you could have used for something else. If you run twelve tests a year and three or four come back inconclusive, that's a quarter of your annual capacity gone. Choosing badly is expensive, which is why prioritising matters more at small scale than at large.

You also give up revenue on purpose. For as long as a test runs, half your visitors see the weaker variant. If the variant is 20% better and runs for a month, you lost 20% of that month's conversions on half your traffic. That's the price of knowing instead of guessing, and it's real money. A holdout costs you in exactly the same way, deliberately, and it's still worth running once or twice a year.

Finally, complexity builds up. Twenty live modifications on your site add a little friction to every future change. It never shows up as a line item, yet it's what makes year three slower than year one if nobody keeps the register of what's live up to date.

Seen against those costs, a platform is worth whatever share of the expensive work it takes off your hands.

The biggest saving is developer time on variants. If building a variant needs an engineer, the real cost of each experiment includes their week, and the number of tests you can run is capped by their backlog. No-code experimentation is what removes that cap.

Next is statistical expertise: sample sizing, significance, correcting for multiple variants, watching guardrails. You can hire it, learn it, or have the tool do it, and the tool is the only one of the three that doesn't cost you months.

Then the discipline not to peek. A tool that holds back the verdict until there is enough sample and clear evidence prevents the most common and most expensive mistake in testing, and that's worth more than most rows on a feature comparison table.

At your size you don't need to pay for multi-team workflows, approval hierarchies, server-side experimentation for apps you don't have, or the long tail of enterprise integrations.

Start with the revenue that flows through the pages you would test. A realistic, well-run small programme might add 10–20% cumulatively in its first year. That figure should come from comparison with a holdout, since adding up individual wins overstates it by a lot.

Set against it the licence, four to eight hours a month at whatever your time is worth, and the revenue given up while tests run.

Say a brand puts €200,000 a year through its product pages. A 15% improvement is €30,000, which comfortably covers a licence and the hours. At €40,000 a year, the same 15% is €6,000, and the sums get much tighter; the hours alone could cost more than that.

That second case is real, and we'd rather say it plainly. Below a certain level of revenue, a formal experimentation programme costs more than it returns. The better move then is to fix what's obviously broken, talk to customers and come back when there's more traffic.

Without experiments you still change your website. You just change it based on opinion, and you never find out which changes did damage.

A brand that redesigns its product page on instinct and loses 8% of conversions will carry that loss for as long as the page stays up, because nothing is measuring it. When you compare the cost of a programme with doing nothing, that's what doing nothing actually means, and it belongs in the same spreadsheet as the licence and the hours.