Skip to main content
Back to insights

The essential guide to A/B testing in email marketing

September 30 - 2022

Email Marketing Strategy 5 min read

5ec3d3ec4e742fa41287f020 20

The definition of insanity, as the old adage goes, is doing the same thing over again and expecting different results. 

What’s true in life also rings true in email marketing. If you keep doing the same things over and over again while expecting different results, you’ll be going nowhere fast.

Naturally, this begs the question: “What should we do differently?” 

It’s here that email marketing A/B testing becomes your best friend. 

What is A/B testing in email marketing?

In order to know what to change, you need to know what works and what doesn’t. As tempting as it is to use gut feel and intuition here, nothing beats a conclusive, evidence-based approach. 

This is what A/B testing gives you.

At its core, A/B testing (or split testing as it’s sometimes known) is an experiment where you split your audience to test different versions of your email marketing campaign in order to see which one performs better.

To run a successful A/B test, you need to create two different versions of one campaign, with changes to a single variable. As you’ll see later, there’s a huge number of variables you can change including everything from the subject line and the preview text through to images and call-to-action buttons.

Once you’ve created the two different versions, you send them to the same sample size of your subscribers, see which one performs better and then send this version to the rest of your database. 

Let’s take a look at an example to understand it further…

A luxury hotel wants to test personalisation in the subject lines of their email campaigns.

Version A has the subject line “A luxury stay is just a click away…” while Verison B’s subject line is “Rachael, your luxury stay is just a click away…”

Version A is sent to 10% of the database and Version B is sent to a different 10% of the database.

The hotel has chosen the open rate as the metric to measure.

Version A had an open rate of 20% while Version B (with the personalisation) doubled this with an open rate of 40%. In this case, Version B is the clear winner so it’s this version that is sent to the rest of the database.  

Why it’s important to A/B test email marketing campaigns

Testing ensures that all that effort of creating and sending emails is maximised to provide ever improving results. Having a structured and intelligent approach to your testing will ensure that effort is focussed in the areas that will make the most difference quickest.

With this approach, the benefits include:

1. Improved conversions and engagement

It’s simple, send out better email marketing campaigns and you’ll see improved conversions and higher levels of engagement. Testing can help you identify what kind of content delivers the best results meaning you can then double-down on this to take your email marketing to the next level.

2. Better understanding of your subscribers

By testing regularly you’ll get a much better understanding of your subscribers. You’ll learn what they are interested in, what makes them tick, what they engage with and much more. By understanding them to a better degree, you can build stronger relationships with them. It’s a win-win and means you can significantly enhance your user experience.

3. A higher ROI

Testing will give you more bang for your buck. Creating great email marketing campaigns has both financial and time costs associated with it. You’ll see a higher return on this investment with a strategic approach to testing.

4. Data-driven decision making

It’s easy to fall into the trap of making decisions based on hunches or gut-feeling. By testing what works and what doesn’t you can make your decisions based on facts and data. 

5. Constant evolution

Being disciplined in testing and recording results ensures that over time you can track what has been tested and why your current approach exists. In a year’s time will the same team be in place? A clear record of past tests will really help your future team.

And, if you share the results across your business, what you learn about your subscribers' preferences in email may well benefit your other marketing channels and sales teams as well.

What to test and what to measure

Earlier, we saw a simple example of testing but there’s a huge number of variables you can test and, more often than not, the more sophisticated your testing, the better your results will be and the more you’ll learn about your audience.

Here are some example of the different kind of tests to carry out:

  • Use of animation
  • Use of personalisation
  • Tone of voice
  • Customer segments
  • Dynamic content
  • Geo-specific content
  • Short Vs long form content
  • No hero image / big hero image
  • Video and interactive content
  • Countdown timers and limited time offers
  • Inclusion and position of social icons and social share
  • CTA buttons – look, position, quantity

Before you carry out any testing, you need to set your objectives. There’s a number of different approaches to objective setting (such as the SMART system), the trick is to find what works best for you. For instance, is the objective of the campaign to increase engagement or is it to drive sales? The more specific you can be with your objectives, the easier it will be to measure success. 

On the measuring front, there’s a huge amount of metrics and KPIs (key performance indicators) you can track. Of course you can look at opens and clicks but it often pays to go beyond this and look at metrics like conversion rates, total revenue, average order value, and more.

According to Marketing Sherpa, subject line was the most common test that email marketers carried out, but was in fact the least impactful. 72% of the respondents consistently deployed tests on subject line but reported that this test only created a strong improvement 35% of the time. 

In contrast, audience segmentation and landing page were only tested 30% of the time but created strong improvement in over 40% of cases.

More complex testing is tricky and can take time to do right if you’re not experienced in setting up the tests which is why it’s a good idea to turn to the experts to help.

Take your A/B testing to the next level

Testing sits at the heart of what we do at Jarrang. We’re committed to ensuring our clients see the best ROI and are always improving their email marketing programs. 

Testing takes time and effort to get right which is why we’re here to help. We can turbocharge your testing by helping you find the variables to test while measuring and analysing the reach, impact, conversions and engagement of your campaigns. 

If you feel like you’re flying blind with knowing what works with your email marketing and what doesn’t or would like to run more tests but are unsure of the best way of doing it, then we’ve got your back.

Our expert team works hand in hand with you and uses our cutting-edge testing platform to identify the A/B tests we know will have the biggest impact on your email marketing program.

If you’d like to find out more, we’d love to talk to you.

Subscribe to our insights newsletter

Be the first to hear about what’s hot in e-marketing and straight to your inbox.

Share this story

Related insights

NB202110 00044 scaled

Mastering Email Marketing Drip Campaigns

Read post
User generated content

Email Marketing Trends For 2025

Read post
5ec3d7d4293839a4d62a6dbf 80

The Role of Email In Your Marketing Communication Plan

Read post
View all insights