Plan, Run and Analyse an A/B Test on Your Website

Stage 1

Decide what to test

Good tests come from evidence, not opinions.

1. 1

   Look for evidence:

   • Heatmaps and recordings (SOP 049): ignored buttons, people not scrolling, form drop-offs.
   • Customer questions in WhatsApp, calls or reviews that the page doesn’t answer.
   • A recent change you think hurt (or helped) conversions.
2. 2

   Common things to test: headline, call-to-action text, form length or position, hero image or video, trust elements (reviews, logos), pricing display.

   Watch out

   Test one element at a time. Change the headline and the form together and you won’t know which one made the difference.

Stage 2

Write the test plan

1. 1

   In a test log sheet, add a row with:

   • Test number (e.g. 001) and owner.
   • Hypothesis: “Changing [X] to [Y] will increase [metric] because [evidence].”
   • Page URL and audience (e.g. all mobile visitors from India).
   • Main metric (one!), plus up to two secondary metrics.
   • Screenshots of version A (control) and version B.
2. 2

   Check traffic: use a sample size calculator with your current conversion rate and the smallest uplift worth detecting (the minimum detectable effect, usually 10–20% relative), at 80% power and 95% significance. If you’d need months to reach the sample, don’t run it on the site; test the idea in ads instead.

   • Worked example: a 3% enquiry rate and a 15% relative uplift (3% → 3.45%) need about 24,000 visitors per version. Many Kerala small-business sites can’t reach that in a month, and that’s an honest answer to give the client.

Stage 3

Build and launch the test

1. 1

   In your testing tool, create an A/B experiment for the page URL and name it after your test number, e.g. “Exp001 – Mobile sticky CTA”.

2. 2

   Create the variant with the visual editor (or ask a developer for bigger changes).

3. 3

   Set the goal to your main metric (e.g. form submit or purchase) and traffic split to 50/50.

4. 4

   Preview both versions on mobile and desktop, in Chrome and Safari, then start the test.

   Tip

   Write the start date in your log. Don’t make other changes to that page while the test runs.

Stage 4

Read the result honestly

1. 1

   Wait at least two full weeks (to cover weekday and weekend behaviour) and until you reach your planned sample size.

2. 2

   Keep version B only if it wins by the decision rule you wrote before launch, e.g. p < 0.05 for a classic test, or “chance to beat baseline” ≥ 95% in a Bayesian tool like GrowthBook or PostHog.

   • Don’t stop a classic test early just because it looks good on day 3 (peeking). Only sequential-testing tools are designed to be checked during the test.
   • Check the split is really about 50/50. A big imbalance (sample ratio mismatch) means the test is broken.
   • Exclude your team’s visits, and avoid launching during festival sales or ad spikes.

   Watch out

   Don’t stop the test the day B looks ahead. Early leads often disappear. That’s called peeking, and it leads to false winners.

3. 3

   Log the result, the numbers and what you learned, even if B lost. A losing test still teaches you something.

4. 4

   Roll out the winner on the real page, then pick the next hypothesis.