Your first A/B test on Shopify, in one afternoon

Shopify brands usually hit the same wall. The theme is a purchased template nobody on the team fully understands, editing it means touching Liquid, and every testing app in the store adds a script that shows up in your Core Web Vitals. So the testing never starts.

This walks through a real first experiment end to end. Budget an afternoon.

Before you touch anything: pick the page#

Open your Shopify analytics and find your single best-selling product page by sessions. Not the collection page, not the homepage — one product URL.

That page is where your first test goes, for the reason covered in our sample-size guide: concentrating traffic on one comparable population is what makes a result readable at small scale. If your top product has 150 sessions a day, you have 75 per variant, and you can detect a large effect in about three weeks.

Write the hypothesis in one sentence#

For a first test on a Shopify product page, the changes with a realistic chance of producing a detectable effect are structural. Some that regularly work:

• Moving the shipping and returns information above the fold instead of inside a collapsed accordion.
• Leading with review count and rating rather than with the product story.
• Replacing a single "Add to cart" with a size-first flow on apparel.
• Making the add-to-cart button persistent on mobile.

Pick one. Write it as: for [segment], [change] will increase [metric] because [mechanism].

We'll use: "For mobile visitors, making the add-to-cart button persistent will increase add-to-cart rate, because on our theme the button scrolls out of view before the reviews."

Connect the store#

In Bixlow you add your storefront domain and paste one script tag into your theme's theme.liquid, once. That's the whole installation — the loader is a single small file served from a CDN, it fetches asynchronously and it fails silently if it can't reach us, so a bad network never takes your storefront down with it.

One script, once, no per-experiment theme edits afterwards. Every subsequent test is published from the Bixlow side without touching Shopify again.

Describe the change instead of building it#

Open the chat and describe the variant in plain language: "on the product page, on mobile only, make the add-to-cart button stick to the bottom of the screen once the original button scrolls out of view".

Bixlow generates the workflow: the selector for the button, the condition for mobile, the variant behaviour and the events to record. You get a visual blueprint you can read and edit — not a black box. If the selector it picked is wrong because your theme names things oddly, you change it there.

This is the step that normally requires a developer, and it's the reason first tests never happen at small brands. Removing it is most of the value.

Set the audience, the metric and the guardrail#

Audience: device is mobile. That's it for a first test — don't stack five conditions, because every condition shrinks your sample.

Primary metric: add-to-cart. We're measuring the event closest to the change.

Guardrail: revenue per session. If the sticky button lifts add-to-cart but the extra carts don't convert, you need to see that in the same view, not discover it next quarter.

Traffic split: 50/50. Uneven splits are for risk management on big changes; for a first test you want the fastest possible result.

Bixlow now shows you the expected duration. If it says 19 days, that's your answer — put it in the calendar and don't touch it.

Validate on staging, then publish#

Preview the variant against your real storefront before it goes live. Check it on an actual phone, not just the responsive simulator: sticky elements are exactly the kind of thing that looks fine in Chrome devtools and covers the price on a real iPhone.

Then publish. The experiment starts assigning visitors immediately, and each visitor stays in their assigned variant for the whole experiment — so a customer who browses on Tuesday and buys on Friday sees a consistent store, which is both better UX and a requirement for the numbers to mean anything.

What to do for the next three weeks#

Nothing. That's the discipline.

Don't check daily. The graph will cross and re-cross, one variant will look like it's winning on day four, and if you act on that you've made a decision based on noise. Bixlow won't show you a winner until the declared sample is reached, which is there specifically to protect you from yourself.

When it finishes, write one sentence about what you learned about your customers — not about the button. "Our mobile buyers read reviews before deciding" is reusable. "The sticky button won" is not.

Key takeaways#

• One script in theme.liquid, once. Every later experiment publishes without touching Shopify again.
• Run your first test on your single highest-traffic product page, not across the catalogue.
• One audience condition, one primary metric, one guardrail. Stacking conditions shrinks the sample you need.
• Preview sticky mobile elements on a real phone, then leave the test alone until it reaches its declared sample.