A/B Testing
A/B testing is an experimentation method that compares two versions of a web page or an email to work out which one produces better results.
Why it matters
A/B testing is the only reliable way to optimise a website. Without testing, you are optimising blind. With A/B testing, every decision rests on data, and the improvements compound to double or triple your conversion rate within a few months.
What is A/B testing?
A/B testing (or split testing) is a scientific method for comparing two versions of the same element to find out which performs better. You create version A (the control) and version B (the variant), which differ by one single element (headline, button colour, image, CTA wording). Traffic is split at random between the two versions, and you measure which one achieves the better conversion rate.
A/B testing replaces opinions ("I prefer the red button") with data ("the red button converts 23% better than the blue one"). It is the most reliable way to optimise a website, a landing page or an email campaign.
How to run an A/B test
Step 1: identify your hypothesis. Write a testable hypothesis: "Changing the headline from 'Our services' to 'How we can help you' will lift the click rate by 15%". Base your hypotheses on data (heatmaps, analytics, user feedback), not on hunches.
Step 2: build the variant. Change one element at a time. If you change the headline AND the image AND the CTA, you will not know which change produced the effect. Multivariate testing (MVT) tests several elements at once but needs far more traffic.
Step 3: split the traffic. Your A/B testing tool (Google Optimize, VWO, AB Tasty) splits traffic 50/50 at random between the two versions. Each visitor sees only one version for the whole duration of the test.
Step 4: wait for statistical significance. This is the most important rule and the one most often broken. You need to reach a 95% confidence level before declaring a winner. In practice that means at least 1,000 visitors per version to detect differences of 5% or more. Stopping a test too early leads to wrong conclusions.
Step 5: analyse and roll out. If variant B wins with 95% confidence or more, roll it out. Document the results and move on to the next test. Optimisation is an iterative, continuous process.
The elements worth testing first
The headline (H1) has the greatest potential impact. Changing a headline can move the conversion rate by 10 to 50%. Test the angle (benefit versus feature), the length, and the tone (formal versus conversational).
The CTA (button wording and colour) is the second most impactful element. "Get my free quote" versus "Send" can make a difference of 30% or more. Button colour (contrast), size and placement are all testable too.
Social proof (testimonials, client logos, numbers) has an impact of 15 to 30% on conversion. Test with versus without social proof, the type of proof (written testimonial versus video versus logo), and the placement.
The form affects conversion directly. Test the number of fields (3 versus 5), the format (one step versus multi-step), and the labels (explicit labels versus placeholders).
The different types of experiment
Not all optimisation tests are alike, and picking the right format saves you weeks of wasted traffic. The classic A/B test compares two versions on a single variable: it is the format best suited to small businesses and to sites with fewer than 20,000 visitors a month. The A/B/n test compares three or more variants at the same time (three different headlines, for example), but it needs proportionally more traffic to stay reliable.
The multivariate test (MVT) tests several elements and their combinations at once (headline, image and button) in order to find the winning interactions. It is powerful but reserved for very high traffic sites, because the number of combinations grows fast. Finally, the redirect test (split URL) compares two pages hosted at different addresses: ideal when you want to pit a complete redesign against the current version rather than a single detail.
For a tradesperson or a sole trader, the rule is simple: start with A/B tests on one high impact element, and only move to multivariate testing when your traffic genuinely allows it.
Classic A/B testing mistakes
Stopping the test too early: you see a 20% difference after 200 visitors and draw a conclusion. Wrong. With so little data, the difference may be down to chance. Wait for statistical significance.
Testing too many elements at once: if you change the headline, the image, the CTA and the colour, you do not know what worked. One change per test.
Not segmenting the results: a test can win overall and lose on mobile. Analyse the results by device, by traffic source and by audience segment.
Not documenting anything: without documentation you retest the same things and lose what you learned. Keep a record of every test with its hypothesis, result and lesson.
Your checklist before launching a test
Before you put a test live, check these points to avoid skewed results:
- A clear hypothesis with a number in it is written down. - Only one element differs between version A and version B. - The conversion goal is properly defined and correctly tracked in your analytics tool. - The minimum sample size is calculated in advance (at least 1,000 visitors per version). - The test is planned to run for at least two full weeks. - You are not checking the results every hour, so you do not conclude too soon.
A/B testing tools
Google Optimize (now replaced by solutions built into GA4), VWO (Visual Website Optimizer), AB Tasty (a French solution) and Optimizely are the most popular tools. For email, most platforms (Brevo, Mailchimp) include A/B testing natively.
A/B testing for a small local business
A local shop or a service provider does not need thousands of tests to improve. The point is to focus the effort where traffic turns into customers: the contact form, the "Book an appointment" button, or the homepage headline. A concrete example: a physiotherapy practice that tests "Book an appointment" against "Reserve my session" may see bookings rise by 20%, simply because the second button speaks more directly to the visitor.
When traffic is low, favour qualitative methods (customer feedback over the phone, watching people use the site on mobile, heatmaps) and apply proven best practice rather than waiting months for a statistical significance you will never reach.
At ConvertiLab, we build A/B testing into our continuous optimisation process to improve the performance of every site and landing page we create, with an approach matched to each client's real traffic.
Practical examples
A SaaS company tests two landing page headlines, 'Automate your tasks' versus 'Save 5 hours a week': the benefit led version wins with 42% more conversions.
An e-commerce site tests a 1 step checkout form against a 3 step one: the 1 step form lifts conversions by 18% on desktop but reduces them by 5% on mobile.
A services site tests adding a video testimonial to its landing page: the contact form conversion rate rises by 32%, which justifies the investment in video content.
Frequently asked questions
How much traffic do I need for an A/B test?
A minimum of 1,000 visitors per version to detect meaningful differences. For tests with a low conversion rate (under 2%), you need 5,000 to 10,000 visitors per version. Use a sample size calculator, such as VWO's, to estimate how long your test will run.
How long should an A/B test run?
A minimum of 2 weeks, even if statistical significance is reached earlier. That covers a full cycle, weekdays and weekends, and avoids seasonal bias. Most tests need 2 to 4 weeks for reliable results.
Is A/B testing useful for small sites?
If your site gets fewer than 1,000 visitors a month, classic A/B testing is difficult, because reaching statistical significance takes months. Favour qualitative testing (user testing, feedback, heatmaps) and apply proven best practice.
Need help with A/B Testing?
Our experts work with you to put an effective strategy in place. Get a free, tailored quote within 24 hours.
Go further
Related terms
Other definitions
Last updated: 6 April 2026
