Prioritizing tests with the ICE framework

When a team starts generating test ideas, the problem stops being a lack of ideas and becomes an excess of them. You have more hypotheses than you can run, and you need to decide which to test first. Doing that on gut feeling, or based on whoever shouts loudest in the meeting, leads to wasting traffic on low-value experiments.

The ICE framework is one of the simplest and most widely used ways to bring order. It scores each idea against three criteria, calculates a number and gives you an objective ranking to decide the order. It's not an exact science, but it structures the conversation and reduces the weight of egos. In this article you'll see how to apply it well.

ICE stands for Impact, Confidence and Ease. For each test idea, the team assigns a score, usually from one to ten, on each of the three criteria. The average or the product of the three gives the ICE score, which ranks ideas from highest to lowest priority.

Impact estimates how much the metric will move if the hypothesis is right. Confidence reflects how much evidence backs that it will work, based on data, not wishes. Ease measures how fast and cheap the test is to run, factoring in design effort, development and the traffic required. A no-code tool like Bixlow raises the ease score of many ideas, because it removes the dependency on development.

The value of ICE depends on being honest when you score. For the ranking to be reliable, it helps to follow a few rules that keep scores from inflating or turning arbitrary.

• Score confidence with real evidence, not with your enthusiasm for the idea.
• Do it as a team and discuss big differences in judgment before averaging.
• Use a consistent scale and shared references so a seven means the same thing to everyone.
• Review and re-rank the list periodically, because the evidence changes.

ICE is a guide, not an oracle. Its scores are subjective and can be gamed, consciously or not, to justify a favorite idea. Use it to structure the discussion and discard what's obviously weak, but leave room for judgment: sometimes it's worth testing a risky, low-confidence bet because of its huge upside. The framework ranks; it doesn't decide for you.

• ICE scores each idea by impact, confidence and ease to prioritize your tests.
• Score confidence with real evidence and do it as a team to reduce bias.
• Treat ICE as a guide to structure the discussion, not as an automatic decision.