What is A/B testing?

A/B testing, defined

A/B testing is a method of comparing two versions of a web page by splitting visitors between them at random and measuring which one gets more conversions. Version A is usually the current page and version B has one change. It shows whether a change actually works, rather than relying on opinion.

A/B testing means showing two versions of a page to different visitors at the same time and seeing which one converts better. Half see version A, half see version B, and the numbers decide.

It’s also called split testing. It’s one of the main tools in conversion rate optimization, because it replaces “I think” with “we measured”.

How an A/B test works

Pick one page and one goal, like demo requests.
Make a version B with a single meaningful change, such as a new headline.
A testing tool splits visitors at random between A and B.
Each visitor keeps seeing the same version on return visits.
Once enough people have seen both, you compare conversion rates.

Changing one thing at a time matters. If version B has a new headline, a new image and a new button, you won’t know which change made the difference.

Statistical significance, in plain words

Any two pages will get slightly different results by luck alone. Flip a coin ten times and you might get seven heads. That doesn’t mean the coin is unfair.

Statistical significance tells you whether the gap between A and B is big enough that luck is an unlikely explanation. Most tools report this as a confidence level, and 95% is the usual bar. In simple terms, it means that if there were really no difference, you’d rarely see a gap this big by chance.

It doesn’t tell you the win is large, or that it will last forever. It just tells you the result probably isn’t noise.

Sample size, and why it matters

Sample size is how many visitors each version needs before you can trust the result. Two things drive it:

Your current conversion rate. The lower it is, the more visitors you need.
The size of the change you want to detect. Spotting a small lift takes far more traffic than spotting a big one.

Work out the sample size before you start, using a free calculator, and don’t stop early. Checking every day and stopping as soon as B looks ahead is the most common way to fool yourself.

A/B testing on low-traffic B2B sites

This is the real constraint for most B2B SaaS teams. A pricing page with a few hundred visitors a month and a low demo rate may need many months to reach a reliable result.

If that’s you, change your approach:

Test big swings, not tweaks. A new offer or a rewritten hero section can show a clear result. A new button colour won’t.
Test higher up the funnel. Clicks on a CTA happen far more often than closed deals, so you reach a result sooner.
Test on your busiest pages. Usually the homepage or a paid landing page.
Sometimes, just ship it. If research shows a clear problem, fix it and watch the trend. Not every change needs a test.

A/B testing vs multivariate testing

An A/B test compares whole versions of a page, usually with one change. A multivariate test changes several elements at once and tests every combination to see which mix works best.

Multivariate tests need much more traffic, so on most B2B sites A/B testing is the practical choice.

Common A/B testing mistakes

Stopping when it looks good. Decide the sample size first and stick to it.
Testing too many things in one variant. You get a result but no lesson.
Running tests for under a week. Weekday and weekend visitors behave differently.
Ignoring lead quality. A version that gets more form fills but worse leads isn’t a win.
Not writing anything down. Keep a log of each test, the hypothesis and the result, including the losers.

Questions about a/b testing

How long should an A/B test run?

Until it reaches the sample size you planned before starting, and for at least one or two full weeks so every day of the week is included. Stopping the moment one version pulls ahead often gives a false result.

What does statistically significant mean in A/B testing?

It means the difference between the two versions is unlikely to be down to chance. Most teams use a 95% confidence level as the bar.

How much traffic do you need for an A/B test?

It depends on your current conversion rate and how big a change you want to detect. Small sites with low conversion rates often need thousands of visitors per version, which is why many B2B pages can only test big changes.