Choosing the primary metric of an experiment

The primary metric is the number that decides whether your experiment succeeded. Singular, declared before launch, and binding. Everything else is context.

Teams resist this. Having one metric feels like throwing information away, and there's always a reasonable case for tracking six. But the moment you have six deciding metrics, you have none — you have a menu, and after the test you'll order from it.

It has to be causally close to your change. If you modified the product page, purchase is causally distant: a lot happens between your change and that event, and most of it is noise you're now measuring. Add-to-cart is closer, so more of the variance you observe is actually attributable to what you did.

It has to happen often enough. A metric that fires 30 times a week will never reach significance at a small brand. This is the constraint that overrides most theoretical preferences.

It has to matter, or be guarded by something that does. A frequent, close metric that doesn't correlate with revenue is a number you can move all day without helping the business. Which is why choosing a close metric obliges you to declare a guardrail.

These three pull against each other. Close and frequent metrics are usually less meaningful; meaningful metrics are usually distant and rare. The primary/guardrail pair is how you resolve it.

You changed the product page. Primary: add-to-cart rate. Guardrail: revenue per session.

You changed the cart or checkout. Primary: checkout completion. Here purchase is close enough to be the primary, and it's frequent enough among people who reached the cart. Guardrail: average order value, because the fastest way to lift completion is to remove upsells.

You changed a landing page for paid traffic. Primary: the next step in your funnel, measured on that campaign's traffic only. Guardrail: cost per acquisition, or revenue per session for that source.

You changed a lead form. Primary: submissions. Guardrail: qualified-lead rate. Non-negotiable — form optimisation without a quality guardrail reliably produces more leads and fewer customers.

You changed navigation or category pages. Primary: product-page views per session, or search-to-product rate. Guardrail: revenue per session. Discovery changes are easy to win on engagement and lose on sales.

Use rates, not totals. "Purchases" goes up if your variant happened to get more traffic that week; "purchases per session" doesn't.

This matters more than it sounds when your split isn't perfectly even, which it never is — assignment is random, so the groups differ slightly, and comparing counts bakes that difference into your result.

"Add-to-cart rate" is ambiguous until you say per what. Per session? Per product-page view? Per unique visitor?

For a product-page change, per product-page view is usually right: it isolates the behaviour of people who actually saw the thing you changed. Per session includes people who never reached the page, diluting your effect with visitors the experiment never touched.

Bixlow makes you pick the denominator at configuration rather than defaulting silently, because the same change can look like +11% or +3% depending purely on that choice — and if you pick it afterwards, you'll pick the flattering one.

One sentence, before any data exists: "This experiment succeeds if add-to-cart rate per product-page view improves, without revenue per session declining."

That sentence is the contract. When the results come in and somebody suggests that actually, scroll depth is up and that's encouraging, you have something to point at.

• One primary metric, declared before launch, binding. Six deciding metrics means none.
• Pick something causally close to your change and frequent enough to reach significance, then guard it with something that matters.
• Always use rates, never counts — your groups are never exactly equal.
• Declare the denominator too. The same change reads very differently per session versus per page view.