Primary and Secondary Metrics: What to Measure in Each Test

Deciding what to measure is one of the most important choices in a test and one of the most often skipped. If nobody says up front which metric decides the winner, it gets chosen afterwards, usually by looking at whichever number came out best. That's one of the quieter ways of fooling yourself with data, because it feels like analysis while you're doing it.

A well-designed experiment separates three kinds of metric: the one that decides, the ones that explain, and the ones that catch collateral damage. Here's what each does, how to choose the primary metric, and why the others matter.

This is the only metric that says whether the variant won or lost. You define it before launch and you don't touch it afterwards. Ideally it sits as close to business value as you can get, a purchase or a sign-up, rather than an intermediate step like a click. A variant that gets more clicks that never turn into anything hasn't improved much.

Sometimes the business event is rare and your traffic won't detect a change in it within any sensible time. Then it can make sense to use a more frequent proxy as the primary metric, on the assumption that it tracks the final outcome closely. Say purchases are too few, so you decide on add-to-cart instead. That's a fair compromise as long as you remember that improving the proxy doesn't guarantee you've improved the business.

Around the primary metric you watch others that add context and protection. They never pick the winner, but they stop you celebrating a result that has quietly caused trouble elsewhere in the funnel or the business.

• Secondary metrics help explain why a variant won: scroll depth, time on page, intermediate clicks.
• Guardrail metrics check that nothing important broke: average order value, return rate, customer satisfaction.
• Counter metrics catch gains made at the expense of something else, such as more sign-ups of worse quality.

The primary metric decides and the others inform. If the variant wins on the primary but makes an important guardrail metric worse, you don't have a clean winner, and you should find out what's going on before rolling it out. The other temptation is promoting a secondary metric to primary after the fact because it happened to look good; at that point you're hunting for a winner in a test that didn't produce one. Write the hierarchy down before the test starts, so that when the numbers come in there's nothing left to choose.