The Why and How of A/B Testing Direct Mail


If you’ve ever managed an online campaign, you know the importance of running A/B or split tests to compare how your target audience responds to different versions. Because the initiative is digital, it’s often simple to make small changes to websites, emails, banner ads, etc. that can take effect almost immediately.

However, that same principle of A/B testing can apply to direct mail campaigns as well. All companies who run direct mail campaigns should be A/B testing their message, audience, and timing. Without insight into these variances, it is incredibly difficult to increase your ROI and you could make the argument that any improvements were the result of pure luck or coincidence.

A/B testing can uncover significant opportunities to enhance future campaigns. It is not a one-and-done activity. It is an ongoing test that continues to evolve as your campaigns evolve. However, many people avoid doing these tests for fear they’ll be complicated or they simply don’t know where to start.

Let’s start with the basics. In an A/B test, everything about your direct mailers is the same except for one element. This could be the list, the offer, a photo, the headline, etc. The most common variables are the offer and list because they can provide the most immediate insight and are fairly easy to produce and analyze. Split your mailing 50/50 and send half of the mailers to one group and half to the other. Then measure which version had a better response rate to determine which variable performed better.

Keep the following tips in mind when starting an A/B test of your own:

Establish a measurable goal

Determine the goal of your mailer and how you will track it. Will you be measuring the number of visits to your website, number of sales, or some other metric? Without having a goal in mind, it will be difficult to know which variables to test.

Don’t test too many variables at once

To truly know how the variable element affected the response rate, it’s recommended you only test one element at a time. Changing mailers drastically would make it near impossible to determine which element contributed to the end result. With a simpler A/B test, calculating the “winning” variable is fairly straightforward.

With very careful planning and tracking, however, some companies choose to test 3 or 4 variables at once. In this case, your test would become an A/B/C or A/B/C/D test. Doing much more than this can seriously dilute the results, making the whole test invalid.

Have a reliable system to track results

When it comes to analyzing the results, the higher the mail quantity, the higher confidence you will have and the smaller the margin of error. A test group of 1,000 will yield less information than a test group of 100,000. If your results come back inconclusive, try testing the same elements a few times till you get a definitive winning variable.

Be sure to track the response rate via coupon redemptions, website visits from QR codes, phone calls, or other reasonable means that align with the goal. You may have 5,000 website visits from mailer A and 7,000 visits from mailer B, but if your metric was number of conversions, this data will not be helpful.

Also, keep in mind how other elements outside the direct mailer may impact the results. Perhaps the landing page on your website needs to be better optimized for conversions. While this, too, can be A/B tested, you may want to consider running a separate A/B test on your landing page at a different time than your mailer so you can get the most reliable data from your efforts.

With A/B testing, the possibilities are endless. Once you find a headline that seems to work well, stick with it and start testing another element. Once that element is found to be successful, move on to the next until your mailer reaches a near perfect state. When you feel this has been achieved, start over and continue testing and tweaking, knowing that there will always be something to improve that will help you reach more customers.

Looking to do A/B testing of your direct mail campaigns? Suttle-Straus’ digital printing and variable data technology can make A/B testing a fairly painless process for today’s marketers. To learn more about Suttle-Straus’ direct mail capabilities, contact your sales rep or call 608-849-1000.

Download Now: Variable Data Printing Infographic

About the author

Latest posts

Dawn Burke

Dawn Burke

Dawn was the Marketing Manager at Suttle-Straus. In her role, she managed all aspects of marketing and advertising, from website updates and content creation to conferences and events.
Loading posts...

Topics: Direct Mail