What Is Split Testing?

Split testing is showing two versions of one thing to two similar groups at the same time, and letting those people decide which one works best. Not your opinion. Not your marketer’s opinion. What your customers actually do. I covered it on episode 127 of the Mentor Business Podcast, the last part of our marketing series, because I see so many owners making big decisions on gut feel.

[embed]https://www.youtube.com/watch?v=5RJ42Me8JnE\[/embed\]

Here’s why it matters. When Microsoft tested its own ideas properly, only about one in three actually worked (Kohavi et al., 2012). Smart people, big budgets, and yet two out of three ideas didn’t move the number they were meant to move. So here’s the question for you as an owner: how many of your ideas are working, and how would you even know?

Split testing isn’t only a marketing thing. You can test a new management structure, a new piece of software, a new offer in a new market. But in this article I’ll keep it to marketing, because that’s where most owners are spending money without knowing what’s working.

Is split testing the same as A/B testing?

Yes, in practice it’s the same idea with two names: version A against version B. Some people say “split test” when the two versions are whole different pages or adverts, and “A/B test” for one small change. It doesn’t really matter what you call it. What matters is that you change one thing at a time.

Here’s where people go wrong. If you change too much between version A and version B, how do you know what made the difference? Say you send two emails and change the headline and the colour. One wins. Was it the headline or the colour? You’ll never know.

So keep it granular. Test the subject line on its own. Test the images on their own. Test whether you use their name in the first few words. Even test a green call-to-action button against a yellow one. That won’t change your open rates, but it will show up in how many people click through, and that’s what tells you what actually works.

What is A/B testing in marketing, and what should a small business test first?

In marketing, A/B testing means testing the things that decide whether someone buys: the offer, the price or how it’s presented, the headline on the page they land on, and the email subject line. For a small business, start big. Test bold differences where the gap is obvious, then refine the winner.

The biggest mistake I see isn’t a technical one. It’s deciding on opinion. The number of times an owner has said to me, “Lewis, email marketing doesn’t work.” Where did that come from? “I get loads of emails and they annoy me.” Then they send one email to a list they bought, nothing comes back, and that becomes the proof.

If you decide something doesn’t work, you go looking for evidence that it doesn’t work. Then you stop doing it, and you miss out on sales. What the data would actually tell you is that that email didn’t work. So try a different one. A different list. A different offer.

The second mistake is frequency. You can’t split test if you only do the activity once a month. If you give something 90 days but only send three emails, that isn’t enough data to tell you anything. It’s why a lot of my own TikTok posts look the same: we’re testing which version gets the views, which one resonates, and which one actually brings in a client.

And watch the headline. When I built MentorBusiness.com I called it “the Netflix of business mentoring.” I didn’t have to explain it was a streaming platform; one word did the work. Your headline has to speak to the person’s problem in the same way, and the only way to find the one that does is to test it.

Big companies work this way too. The latest iPhone looks almost exactly the same as last year’s. Why? Because they tested different versions years ago and now they know which one people buy. They refine what works rather than starting again.

How do you do a split test?

You don’t need software to start. Open a spreadsheet, write down what winning means and which version you think will win, change one thing, run both versions at the same time, and keep going until the data shows a winner. Most email and ad tools, and a good CRM, will run split tests for you once you’re ready.

1. Decide what winning means before you start. More sales, more enquiries, more replies? Not more likes. Likes don’t pay the bills. Sales do.
2. Write down which version you think will win. That’s the top of your spreadsheet.
3. Change one thing. Just one.
4. Run both versions at the same time, to two similar groups.
5. Keep going. You might run the same change for a month and send 12 or 15 emails before you judge it. It depends on what you’re selling.
6. Keep the winner and test again off the back of it.

One more thing. Don’t hand this straight to your marketer and walk away. They’ll run it based on what they already know. It’s you as the owner who needs to understand this, so you can ask the right questions and spot when someone is showing you nice-looking posts instead of results. The more you learn, the more you earn.

How many people do you need for a valid A/B test?

Start with whoever you’ve got: 10 people or 100. A small list isn’t a reason not to test. With only 10 people you won’t see small differences, so test bigger changes, such as a different offer. Once you have hundreds or thousands, smaller differences will start to show.

I say this because the size of the list is the easiest excuse there is. “This doesn’t work for me, my list is too small.” You don’t need a bigger audience to start learning. You need to start.

And please don’t say, “I gave it three months and it didn’t work.” That’s like saying physics doesn’t work. A/B split testing isn’t a marketing strategy. It’s science. You’re collecting data, and data is what lets you scale. Try walking into a bank to borrow money to grow without it.

Your one job this week

Pick one thing: your price, your offer or an email subject line. Write down which version you think will win. Then test it.

Want a second pair of eyes on your marketing?

If you want a deep dive on your marketing — what offers you’re putting out there, who your target market is, and whether you’re even pricing correctly — book a virtual coffee with me. My prices are on the site, so there are no surprises.

Next on the Mentor Business Podcast: what does a business coach actually do?

References:

Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 786–794. https://doi.org/10.1145/2339530.2339653

Listen or watch on your favourite platforms

Listen to the full episode: What Is Split Testing? A/B Testing for Business Owners — Mentor Business Podcast, Episode 127

Listen on Spotify

Watch on YouTube

Listen on Apple Podcasts

---