The Email Savant Series: Simplifying A/B Split Testing

Written by Mark Vogel

June 14, 2018

Are your email marketing messages returning the best results possible? Are you sure? The only way to know is to test – and do so repeatedly on an ongoing basis. There are two kinds of email marketing tests: A/B split tests and multivariate tests.

A/B split tests compare an original version of your message (known as the “control”) against a second version (known as the “variant”). The variant will include a change to a just ONE element, such as the subject line, an offer, the time of day, a call to action (CTA), a single element of the design, and so on.

Multivariate testing means that you test several elements at once within your email message so that one message will be completely different from the other. While multivariate testing has its advantages in certain situations, there are also disadvantages, including statistically inaccurate results with smaller lists and/or shorter test cycles, and higher levels of complexity in understanding the results. With A/B split testing, results are usually quite clear and overt – which leads to obvious actions to implement in subsequent messages. So, for this blog, we’ll discuss A/B split testing.

Why Should You Test?

There are many reasons to create an A/B split testing effort, and to make it an essential component of your overall email marketing strategy:

• You’ll make decisions about your email marketing campaigns based on data rather than on guesswork or whatever “feels right” to you at the time.
• You’ll be able to measure when messages work and when they don’t, based on your subscribers’ behaviors.
• You can help increase your return-on-investment (ROI) by delivering more effective email marketing campaigns – yielding more opens, more clicks and more conversions.
• You’ll reduce the risk of launching an all-new campaign by testing it before making final decisions.

What Should You Test?

Testing Email Subject Lines

Your recipients won’t click through on a call-to-action – or convert to a sale – unless they OPEN your email! Nearly half of your email recipients will open email based on the subject line, while two-thirds of them might report your message as spam based solely on the subject line. The subject line is the best place to start your testing efforts.

• Keep the subject line short and clear.
• Try personalizing it with the recipient’s first name.
• Test certain words such as “Free” or create a sense of urgency.
• Try asking a question in the subject line and answer it in the message.
• Test some tasteful humor, or appeal to your subscribers’ emotions or pain points.

Testing Email Content

Many of the same tactics for subject lines can apply to the body copy of the message. Personalization is a proven method to boost response rates.

• Appeal to your recipients’ “FOMO” – Fear of Missing Out – and create a sense of urgency or exclusivity. Test this vs. standard features/benefits content.
• Test shorter or bulleted text vs. longer sentences.
• Test variations in the “preheader text” or first sentence – that’s the snippet of text that shows up in the preview pane just below the subject line.
• For content such as a white paper or case study, test the entire content in the body of the email vs. a click through to your website or to open a PDF.
• Test “soft sell” content vs. “hard sell” content.
• Try a “personal tone” in your message vs. a more “businesslike tone,” and test some appropriate humor.
• Try using testimonial-based “voice of the consumer’ copy vs. your own sales and marketing copy.
• Segment your list as much as possible, and test different messages based on the recipient’s profile – customer vs. prospect, if-you-bought-X-then-you’ll-like-Y, and so on.

Testing Email Links

The “paradox of choice” theory says that the more choices we have, the more unlikely we are to make any decision at all. Many marketers want to give their audience all the click choices possible – which may be the wrong decision.

• Try testing several links vs. two – or even just one!
• How many words should you include in link text? MailChimp reports that eight to twelve is optimal. Test different lengths to see what works for you.
• Try out different link designs to see what draws more clicks – bold vs. non-bold, standalone vs. buried in a paragraph, colored text vs. plain black, and so on.
• Test a graphical button for the link vs. text.

Testing Email Design

There are infinite ways to create your email marketing templates. While all messages need to be responsive on multiple devices, here’s a small sampling of layout ideas to consider for testing:

• More images/less text vs. more text/fewer images
• Fonts, point sizes, line heights, font colors
• Dark type against white background vs. lighter type against a darker colored background
• Clean, crisp or “minimalist” design vs. busier or more detailed design for greater excitement
• One-column vs. two-or-more columns
• One product vs. multiple products
• Product images sizes
• Larger header image vs. smaller, or no, header image
• Arrows, borders and other features to guide the eye to a call-to-action
• Test different colors for a call-to-action button

Testing Offers and Calls-to-Action

If the primary reason for our email is for recipients to take a specific action – buy a product, call you, download a white paper, and so on – then you need to understand what offers and calls-to-action will help you meet your goals.

• Test different incentives, such as free shipping, buy-one-get-one, free e-book, future discounts, loyalty points, and so on.
• For a limited-time-offer, test one version with a countdown clock to emphasize the urgency vs. one without.
• Try bundling multiple products into fewer offers vs. long lists of individual products.
• Test the commonly-used “Click Here” or “Learn More” vs. more exciting call-to-action text or buttons such as “Save Now” or “Enroll Today”.
• Place the primary call-to-action at the top of the email vs. at the bottom following the body copy.
• Test a large, graphical “Chat Now” or “Call Now” button vs. burying contact info in the footer of the message.
• Try a $10 off discount vs. a 10% off discount.

Testing Email Timing

Find out when your audience wants to hear from you through testing.

• Send the same message at the same time on two different days to determine the best day of the week.
• Then run the same test to learn what time-of-day is best. That also means testing sends at different times based on recipient’s location – if 8 am Tuesday is best, then try segmenting your list by location so you’re sending at 5 am to those in California, or at 9 pm on Monday if they are in India.
• Test frequency preferences by sending once a week to one group and less frequently to another group. If weekly wins, then test a group to receive an email twice a week vs. once a week, and so on.
• Test “thank you for purchasing” emails with multiple offers of similar products that are sent immediately after checkout vs. a standard receipt email or routine newsletters.

Testing isn’t a once-and-done activity. Continue testing new designs, offers, and calls-to-action. Your target audiences are constantly evolving. Your email marketing strategy must evolve with them!

Mark Vogel is president of Vogel Marketing Solutions LLC and serves as a lead email consultant for Nxtbook Media. He has more than 35 years of experience in the marketing world and has been actively engaged in email campaigns for more than 20 years. His email marketing clients include Fortune 500 companies, e-commerce retailers, non-profits, local businesses, and more. He can be reached at Mark@VogelMarketing.net