A/B testing is a controlled experiment that splits live traffic between two versions of the same page or element and measures which one performs better against a goal set in advance, using a statistical check to separate a real difference from chance.
How to run an A/B test that actually decides: split, sample size, the peeking that skews results, and what Google asks for when testing.
Two identical mugs; one was drunk from
What A/B testing means
An A/B test puts two versions of the same thing up against each other. Version A is usually the current one and version B carries one specific change: a different headline, a different form, a different price on display. Visitors are split at random between them and each group sees only its own version. Because both versions run at the same time and the split is random, everything else hits both groups equally: the season, a live campaign, a piece of industry news.
That simultaneity is what turns a comparison into a test. Without it you have two different periods and no way of telling whether the improvement came from your change or from the market.
The method is not limited to web pages. It works the same way on email subject lines, ad creatives, sign-up flows and displayed prices. What changes from case to case is the volume you need and how fast it builds up.
How an A/B test works
A well-built test always follows the same order, and skipping a step usually invalidates the result.
You start with a hypothesis that can turn out wrong: "moving the form above the fold will increase quote requests". Then you fix a single primary metric, the one that decides, and work out how many visitors you need. That calculation depends on the smallest effect you want to be able to detect: the smaller the improvement you are after, the bigger the sample. Detecting a 20% jump in conversion rate costs a fraction of the traffic it takes to detect a 2% one.
With the sample size settled you launch the random split and let it run. Once you hit the planned volume you check whether the observed difference is compatible with chance. The usual threshold is 95% confidence, which means accepting a 5% probability of declaring a winner that does not exist. That 5% only holds if you look at the result once, at the end.
The last step is the decision: roll out B, keep A, or repeat the test with a bigger sample. A flat result is information too, because it kills an idea before it turns into a project.
About the tools
Google shut down Google Optimize, its free testing tool, on 30 September 2023 and never released a replacement of its own. Anyone starting today picks between third-party tools or splits the traffic on their own server. The second option is the cleanest for SEO, because the variant is served from the same URL and without rendering delays.
The random split and the simultaneity are what separate a test from a before-and-after comparison.How it differs
A/B testing versus other test methods
The most expensive confusion in practice has nothing to do with multivariate testing. It happens when a before-and-after comparison gets taken for an A/B test.
In a before-and-after comparison you publish the change, wait a month and compare it with the previous month. It looks like a test and it isn't. Between the two periods the season also changed, along with the competition, your campaigns and often Google's own algorithm. If sales go up 8%, you have no way of splitting that 8% between your change and everything else. An A/B test solves exactly that, because both groups live through the same month.
Method
What it compares
When it makes sense
A/B test
two complete versions, at the same time, with a random split
one clear change and enough traffic to close the sample
Multivariate test
combinations of several elements at once
plenty of traffic and a need to know which element carries the weight
Split URL test
two different URLs rather than two variants of one
full redesigns; requires a 302 redirect and a canonical
Before-and-after comparison
two consecutive periods, with no control group
last resort when traffic will not support a test
Personalisation
fixed segments, with no control group
adapting the experience, not measuring an effect
A multivariate test tries combinations of several elements at once and splits the traffic across all of them. With four combinations you need roughly four times the sample of an A/B test to reach the same certainty, so on mid-sized sites it rarely closes at all.
Personalisation gets mistaken for testing because it also shows different content to different people. The difference sits in the control group: personalisation assigns by segment and does so permanently, so nobody is left to compare the result against.
Why it matters
The main payoff of A/B testing is defensive. It stops you rolling out losers. A redesign that drops conversion by 5% goes unnoticed for months if nobody measured it against the previous version, and by then the team has already built on top of it.
Most tests come out flat or negative. That is discouraging at first and it is the most useful part of the method: every idea dropped on the evidence is budget you never spend building it.
It also changes how the team argues. A hypothesis written down before anyone sees the data can be judged; an opinion about which headline sounds better cannot. A/B testing moves the discussion from taste to outcome, and that counts most when the decision affects a form the whole business runs through.
Best practices
Fix the primary metric and the sample size before you launch, and write them down somewhere you cannot quietly change them later.
Let the test run to the planned volume and always close on whole weeks: a Monday behaves nothing like a Sunday.
Change one thing per test if you want to be able to explain the result.
If the test uses different URLs, redirect with a 302 and point the variant's rel="canonical" at the original URL. Google asks for it that way.
Take the test setup down as soon as you have made the decision. Google warns that an experiment left running far longer than needed can be read as an attempt to deceive.
Keep the flat and negative tests too. They are the record of what has already been tried and they stop the same experiment coming back a year later.
Common mistakes
Stopping the test on the first day it looks "significant". Every interim peek is another chance for randomness to cross the threshold, and with repeated checks the real probability of a false winner climbs well above the 5% advertised.
Measuring the button click instead of the order. A variant can win on clicks and lose on revenue.
Using a 301 redirect for the test. Google then swaps the original URL in its index for the variant's, and when the test ends you are left ranking the wrong URL.
Serving a different version to Googlebot "so as not to disturb it". That is cloaking and it breaks Google's spam policies.
Launching a test on a page with so little traffic that it will never reach the sample size, and deciding anyway on whatever data turns up.
Manuel Riveiro RodriguezCEO & Digital Strategist
Want to know how this applies to your site? We will go through it with you.
Until it reaches the sample size you calculated at the start, and always closing on whole weeks. Two weeks is the norm on sites with moderate traffic. Ending early because the result already looks clear is the most common way to produce a winner that does not exist.
How much traffic do you need for A/B testing?
It depends on the effect you want to detect and on your current conversion rate. The smaller the expected improvement, the bigger the sample. With few conversions a month it pays to test big changes rather than nuances, or to measure an intermediate goal that happens more often than the purchase.
Does A/B testing hurt SEO?
No, provided you follow Google's guidance: never serve different versions to Googlebot and to people, use a 302 redirect rather than a 301 when the test splits across several URLs, point rel="canonical" at the original and take the setup down when you finish.
What is the difference between A/B testing and multivariate testing?
An A/B test compares two complete versions. A multivariate test tries combinations of several elements at once to find out which of them carries the most weight. Multivariate answers more questions, but it splits the traffic across many more variants and therefore needs considerably more volume.
What do I do if the test gives no clear winner?
A tie usually means the change was too small to matter. Keep whichever version is simpler to maintain and spend the next test on a more ambitious hypothesis. Repeating the same test with a bigger sample only pays off if the observed difference pointed clearly in one direction.
1969The effect of checking results several times during an experiment has been documented since Armitage, McPherson and Rowe, "Repeated significance tests on accumulating data", Journal of the Royal Statistical Society A.
