# A/B testing on Shopify: when Rollouts is enough and when you need more

> What Shopify's native Rollouts experiments do well, where they fall short (no targeting, Grow plan and up), how to install a testing script correctly and how to measure up to the order.

- Source: https://bixlow.com/en/blog/shopify-ab-testing-rollouts-or-more
- Language: en
- Published: 2026-11-02T14:00:00.000Z
- Updated: 2026-11-03T09:15:25.686Z
- Author: fernando
- Topics: A/B testing, Shopify, Shopify Rollouts, ecommerce CRO, personalisation, no-code experimentation
- Publisher: Bixlow

---
Shopify now has its own A/B testing: it is called Rollouts, it lives under Markets > Rollouts, and its experiments are available from the Grow plan up. It is enough when you want to compare two versions of your theme, a checkout configuration or a discount, and measure all the way to the order without installing anything. You need something else when a variant should only reach one campaign or only mobile visitors, when the change is a line of copy or a block that does not justify duplicating the theme, or when you are on the Basic plan. Below I go through what it does well, where it falls short and how to set up the other route without breaking the store.

## What Shopify Rollouts is

Rollouts appeared in Shopify's changelog in June 2026. A rollout is a scheduled set of changes to your main theme, your checkout and account pages, discounts or catalogs. There are three types:

- **Launch:** publishes the changes permanently, on the date you choose.
- **Event:** publishes them for a period and then rolls them back on its own. Built for campaigns like Black Friday.
- **Experiment:** compares a treatment against a control so you can measure the impact before launching to everyone.

Rollouts in general are available from the Basic plan. Experiments only from the Grow plan up. That is stated plainly on the requirements page of the Help Center.

## What Rollouts does well

The best thing about it is that it lives inside Shopify and measures what matters.

- **It tests whole themes.** You can edit the main theme in the theme editor inside the rollout or replace it with another theme. If you are thinking about moving from Dawn to Horizon, you can show the new theme to half your visitors before deciding.
- **It tests the checkout.** It can compare two checkout and customer account configurations, which an external tool cannot touch.
- **It measures to the order.** A theme experiment reports conversion rate, bounce rate, reached checkout rate and add to cart rate. A checkout experiment reports checkout conversion rate. Since the checkout is Shopify's, the order is counted without workarounds.
- **It copies for you.** Creating the rollout makes a copy of the published theme, so you can keep editing the store while the test runs.
- **No extra scripts.** You install nothing and add no weight to the store.

The default split is 50% control and 50% treatment, you can change it, and an experiment ends by default 90 days after it starts.

## Where it falls short

This is what you should know before setting up your first experiment. All of it comes from Shopify's own help pages:

1. **No targeting.** In their words, you cannot target a rollout to a customer segment, a customer tag or a device. Visitors are put in groups at random.
2. **No filtering.** You cannot break results down by device or country, and you cannot create custom metrics.
3. **Only consenting visitors count.** Everyone gets a version, but only visitors who accept tracking are counted in the results.
4. **One theme change per rollout.** Vintage themes and Liquid template changes are not supported.
5. **Online store only.** Headless and custom storefronts are left out.
6. **The statistics are not explained.** The analytics page does not say which method it uses or when it considers a result conclusive. It shows you the metrics and the call is yours.

None of this is a flaw. It is a tool built to test versions of the whole store, and it does that well. The trouble starts when what you want to test is smaller or more specific.

## When a script-based tool makes sense

Picture a natural cosmetics shop in Bilbao that sells mostly through Instagram. These are the tests it would have on its list that Rollouts does not cover well:

- **A message for one campaign.** Someone arriving from the Instagram ad with `utm_campaign=night-routine` should see the night routine on the homepage, not the generic banner. That is [personalisation by audience](/en/platform/personalization), and Rollouts does not target.
- **A mobile-only change.** A sticky add to cart button on the product page makes sense on mobile, not on desktop. The idea is written up in [sticky add to cart on mobile](/en/ideas/sticky-add-to-cart-on-mobile).
- **A line of copy or a block.** Changing "Shipping in 24/48 h" to "Get it on Thursday" should not require duplicating the theme.
- **App widgets.** Reviews, size pickers or "frequently bought together" blocks are often drawn by an app with its own JavaScript. Moving them or changing their heading is easier on the real page.
- **New versus returning.** Someone who has already bought does not need to be told who you are. A first-time visitor does.
- **You are on Basic.** Without the Grow plan, Rollouts has no experiments.

A script-based tool applies changes in the visitor's browser, on top of the page Shopify has already generated. It does not touch the theme or the store's database. That is what Bixlow does: you install one script, the agent proposes hypotheses and builds the variants on the live store, and nothing goes live until you approve it.

## How to install a script on Shopify (and how not to)

There is a lot of confusion here, so step by step.

1. **In `theme.liquid`, inside the `<head>`.** This is the right place for a script that changes the page. In the admin, go to Online Store > Themes, click the three dots on the published theme and choose Edit code. Open `layout/theme.liquid` and paste the script as high as possible inside the `<head>`. If you ever update the theme to a new version, check that the script is still there.
2. **Through Google Tag Manager, if GTM sits in `theme.liquid`.** If GTM is already pasted into the theme, add the script as a Custom HTML tag on all pages. It loads a little later, which sometimes means the change shows up a moment late.
3. **Not in custom pixels.** Under Settings > Customer events you can create custom pixels, but they run in a sandbox. Shopify's own help says they cannot display elements on the page. They are for measuring, not for changing what the customer sees. The same goes for GTM if you added it as a custom pixel: whatever loads there cannot touch the page.

Stores that use apps sometimes switch on "app embeds" in the theme editor. Those exist only for apps. Bixlow has no Shopify app, so the route is `theme.liquid` or GTM.

Then open the store in a private window and check that the script loads on the homepage, a product page and the cart.

## Measuring orders: time to be honest

Shopify's checkout does not load `theme.liquid`. Your theme scripts do not run in the payment steps, and Shopify has been retiring the "additional scripts" that used to be pasted into the thank you and order status pages. Custom pixels do see the purchase event, but they live in a sandbox and do not talk to the scripts on the page.

In plain terms: with a script-based tool, the goal closest to the order that you can measure reliably is reaching checkout. In Bixlow you set it up as a goal on the click on the cart's "Check out" button, or by URL when the visit lands on the checkout address. Add to cart works too, by click or by dataLayer event if your theme already pushes one.

It is not perfect, but it works for most tests that happen before payment. If you change the product page or the cart and more people reach checkout while nothing in the checkout changed, it is reasonable to expect orders to follow. Keep an eye on Shopify's analytics during the test anyway.

If what you are testing has to be judged on orders no matter what (a new theme, the checkout itself, a discount), use Rollouts. That is its strongest card.

## What to test first

If you do not know where to start, this order works for almost any small store:

| Test | Tool | Why |
|---|---|---|
| New theme versus current one | Rollouts | It changes the whole store and you want orders |
| Checkout configuration | Rollouts | An external tool cannot reach it |
| Shipping cost and time next to the price | Script | It is a block, not a theme |
| Sticky add to cart on mobile only | Script | Rollouts cannot target by device |
| Homepage message by campaign | Script | You need the UTM |
| "You're €X away from free shipping" in the cart drawer | Script | The cart drawer is drawn by JavaScript |

Before publishing any test, place a full test order with each variant, on a phone. And work out how long it will take with the [sample size calculator](/en/tools/sample-size-calculator): with little traffic, test big changes.

If you want the full walkthrough of a first test, I wrote it up in [your first A/B test on Shopify, in one afternoon](/en/blog/first-ab-test-shopify-one-afternoon).

## Using both at once

You can, with one rule: do not test the same thing on the same page with both at the same time. If a Rollouts experiment changes the product page and a script test runs on that same page, the results mix and neither means anything.

What I would do: Rollouts for the big theme and checkout decisions, once or twice a year. The script for the day to day, the campaigns and the small changes you want to check every week.

## Frequently asked questions

**Can Rollouts test prices?**
It can include catalog changes, and setting a catalog to active publishes its prices and product availability. It also supports discounts. There are rules (for example, a discount can only be part of one unended rollout at a time), so read the Help Center first.

**Can Bixlow change Shopify's checkout?**
No. Shopify's checkout does not load theme scripts. Bixlow works on the storefront: homepage, collections, product pages and cart.

**How much does A/B testing with Bixlow cost?**
The free plan analyses your site and proposes hypotheses, but does not run tests. A/B tests start on the Standard plan, at €24.17 a month billed yearly.

**How does Bixlow decide a variant has won?**
With Bayesian statistics. There is no winner until a variant has at least a 95% probability of being the best and a minimum sample and duration are met. It also warns you if the traffic split has gone out of balance.

---

## Meet our AI CRO agent

What you just read, put to work by an agent: it understands your visitors, decides what to test or personalize, builds it on your site without a developer and tells you when a result is real.

https://bixlow.com/en/ai-cro-agent