How to Measure the Incrementality of Your Personalizations

A personalization can look like a win and contribute nothing. If you show an offer to high-intent visitors, many will buy, but they might have bought anyway without it. Incrementality measures exactly that: how many extra conversions the personalization generated compared to doing nothing at all. It's the difference between celebrating real results and celebrating coincidences.

Measuring incrementality is what separates a serious personalization program from one based on gut feeling. It takes discipline: defining a control group, keeping it, and reading the data honestly. In this article you'll see how to do it right, which mistakes produce false victories, and how to decide whether an experience deserves to scale or be retired.

What incrementality is and why it matters#

Incrementality is the causal effect of your intervention: the result that wouldn't have happened without it. You don't measure it by looking only at the people who saw the personalization, but by comparing that group with an equivalent one that didn't see it. Without that comparison, you credit your work for conversions that were already on their way.

It matters because without incrementality you don't know where to invest. An experience can have plenty of associated conversions and still a zero incremental effect, while a more modest one might be driving real sales. Measuring the causal effect tells you what to keep and what to discard.

How to set up a control group#

The control is a group of visitors who meet the conditions of the personalization but receive the standard version. It must be assigned randomly and kept in place for the entire experiment. Randomness ensures both groups are comparable and that the difference comes from the intervention, not from something else.

• Assign visitors at random between the personalized group and the control.
• Keep the control in place for the whole test, with no reassigning.
• Make sure you have enough volume to detect the expected effect.
• Define the success metric before you start, not afterwards.
• Set a minimum duration that covers complete purchase cycles.

Mistakes that produce false victories#

The most common mistake is measuring only the personalized group and comparing it against past periods. Seasonality, campaigns, or external changes contaminate that comparison. Another mistake is stopping the test as soon as the numbers look good: halting at a favorable spike inflates results that won't hold up.

It's also wise to distrust tiny segments. With low volume, any difference looks large but isn't reliable. Before declaring a winner, make sure the result is stable and that the gap versus the control is consistent over time.

Key takeaways#

• Measure the causal effect against a control, not the absolute conversions of the exposed group.
• Assign the control at random, keep it in place for the whole test, and give it enough volume.
• Don't stop at the first favorable spike, and don't trust segments that are too small.