The “winning” headline that wasn’t
A Kochi digital marketing institute tested two landing page headlines. After three days, version B was 40% ahead and the team switched to it.
Leads didn’t rise. With only 300 visitors per version, the early gap was mostly chance.
The next test was planned properly: one bold change (fees on the first screen), about 6,000 visitors per version, four full weeks. The winner lifted conversion by 22% and held.
Illustrative example based on a common pattern in Kerala e-commerce accounts. Numbers rounded.
What A/B testing is
An A/B test splits traffic randomly between version A (the current one) and version B (the change), so that the only difference is the change itself.
Small sites rarely have enough traffic to detect small differences. That’s why big, meaningful changes (offer, headline, form length) are better to test than button colours.
Visitors per version ≈ 16 × p × (1 − p) ÷ (p × lift)² (95% confidence, 80% power)
How many visitors and days does your test need?
- Visitors per version
- 6,144
- Total visitors
- 12,288
- Days at current traffic
- 41
Run the test for at least this long and full weeks (to cover weekday and weekend behaviour). Don’t stop early when one version looks ahead.
Approximation for 95% confidence and 80% power. Use a full calculator for high-stakes tests.
Want a second opinion on your numbers? Join the community and post them for doubt-clearing.
Worked example
| Current conversion rate | 4% |
|---|---|
| Smallest lift worth detecting | 25% (4% → 5%) |
| Visitors needed per version | about 6,100 |
| At 300 visitors a day | about 41 days, so run 6 full weeks |
How experienced marketers use A/B test
1. Landing pages
Test the offer, headline, form length or proof on pages with enough traffic.
2. Ads
Use Meta’s A/B test or creative testing and Google Ads experiments to compare versions with split audiences.
3. Email subject lines
Email tools test subject lines on a small share before sending the winner to the rest.
What good looks like
| Business type | Starting rule | Why |
|---|---|---|
| Minimum test length | 2 full weeks | Covers weekday and weekend behaviour. |
| Confidence level | 95% | Standard in most tools. |
| Changes worth testing | Bold, not tiny | Small sites can only detect big effects. |
When the number lies
- Peeking and stopping early
- Checking daily and stopping when one version leads makes false winners very likely.
- Uneven traffic
- If one version gets more ad traffic or a different audience, the test isn’t fair.
- Seasonality
- A festival week can make any version look better. Run tests across normal weeks.
Mistakes to avoid
Testing many changes at once
You can’t tell which change worked.
Ending the test early
False winners that don’t hold.
Testing on pages with too little traffic
Tests take months or never reach a clear answer.
Related terms
Questions people ask about A/B testing
What is A/B testing?
Comparing two versions of a page, ad or email with randomly split audiences to see which one gets better results.
How long should an A/B test run?
Until each version reaches the planned sample size, and at least two full weeks. Use the calculator above.
Is Google Optimize still available?
No. Google Optimize closed in 2023. Use tools such as VWO, Optimizely, AB Tasty or your page builder’s split testing.
Can I A/B test with low traffic?
Only for big changes. With low traffic, test bold ideas, or use qualitative tools like heatmaps and user feedback.
Put it into practice
- SOP 060: Plan, Run and Analyse an A/B Test on Your Website →
- SOP 327: Test Meta Ad Creatives and Find Winners Faster →
Prefer to talk first? Book a free counselling call.