# Your first A/B test on Shopify, in one afternoon > No theme edits, no app that slows your storefront, no developer. From hypothesis to live experiment on a real product page, step by step. - Source: https://bixlow.com/en/blog/first-ab-test-shopify-one-afternoon - Language: en - Published: 2026-10-01T09:00:00.000Z - Updated: 2026-10-06T09:28:18.400Z - Author: fernando - Topics: Shopify, A/B testing, no-code experimentation, ecommerce CRO, new brands, guardrails, cart abandonment - Publisher: Bixlow --- Shopify brands usually hit the same wall. The theme is a purchased template nobody on the team fully understands, editing it means touching Liquid, and every testing app in the store adds a script that shows up in your Core Web Vitals. So the testing never starts. This walks through a real first experiment end to end. Budget an afternoon. ## Before you touch anything: pick the page Open your Shopify analytics and find your single best-selling product page by sessions. Not the collection page, not the homepage — one product URL. That page is where your first test goes, for the reason covered in our sample-size guide: concentrating traffic on one comparable population is what makes a result readable at small scale. If your top product has 150 sessions a day, you have 75 per variant, and you can detect a large effect in about three weeks. ## Write the hypothesis in one sentence For a first test on a Shopify product page, the changes with a realistic chance of producing a detectable effect are structural. Some that regularly work: - Moving the shipping and returns information above the fold instead of inside a collapsed accordion. - Leading with review count and rating rather than with the product story. - Replacing a single "Add to cart" with a size-first flow on apparel. - Making the add-to-cart button persistent on mobile. Pick one. Write it as: *for [segment], [change] will increase [metric] because [mechanism]*. We'll use: "For mobile visitors, making the add-to-cart button persistent will increase add-to-cart rate, because on our theme the button scrolls out of view before the reviews." ## Connect the store In Bixlow you add your storefront domain and paste one script tag into your theme's `theme.liquid`, once. That's the whole installation — the loader is a single small file served from a CDN, it fetches asynchronously and it fails silently if it can't reach us, so a bad network never takes your storefront down with it. One script, once, no per-experiment theme edits afterwards. Every subsequent test is published from the Bixlow side without touching Shopify again. ## Describe the change instead of building it Open the chat and describe the variant in plain language: "on the product page, on mobile only, make the add-to-cart button stick to the bottom of the screen once the original button scrolls out of view". Bixlow generates the workflow: the selector for the button, the condition for mobile, the variant behaviour and the events to record. You get a visual blueprint you can read and edit — not a black box. If the selector it picked is wrong because your theme names things oddly, you change it there. This is the step that normally requires a developer, and it's the reason first tests never happen at small brands. Removing it is most of the value. ## Set the audience, the metric and the guardrail Audience: device is mobile. That's it for a first test — don't stack five conditions, because every condition shrinks your sample. Primary metric: add-to-cart. We're measuring the event closest to the change. Guardrail: revenue per session. If the sticky button lifts add-to-cart but the extra carts don't convert, you need to see that in the same view, not discover it next quarter. Traffic split: 50/50. Uneven splits are for risk management on big changes; for a first test you want the fastest possible result. Bixlow now shows you the expected duration. If it says 19 days, that's your answer — put it in the calendar and don't touch it. ## Validate on staging, then publish Preview the variant against your real storefront before it goes live. Check it on an actual phone, not just the responsive simulator: sticky elements are exactly the kind of thing that looks fine in Chrome devtools and covers the price on a real iPhone. Then publish. The experiment starts assigning visitors immediately, and each visitor stays in their assigned variant for the whole experiment — so a customer who browses on Tuesday and buys on Friday sees a consistent store, which is both better UX and a requirement for the numbers to mean anything. ## What to do for the next three weeks Nothing. That's the discipline. Don't check daily. The graph will cross and re-cross, one variant will look like it's winning on day four, and if you act on that you've made a decision based on noise. Bixlow won't show you a winner until the declared sample is reached, which is there specifically to protect you from yourself. When it finishes, write one sentence about what you learned about your customers — not about the button. "Our mobile buyers read reviews before deciding" is reusable. "The sticky button won" is not. ## Key takeaways - One script in `theme.liquid`, once. Every later experiment publishes without touching Shopify again. - Run your first test on your single highest-traffic product page, not across the catalogue. - One audience condition, one primary metric, one guardrail. Stacking conditions shrinks the sample you need. - Preview sticky mobile elements on a real phone, then leave the test alone until it reaches its declared sample. --- ## Meet our AI CRO agent What you just read, put to work by an agent: it understands your visitors, decides what to test or personalize, builds it on your site without a developer and tells you when a result is real. https://bixlow.com/en/ai-cro-agent