Subject Line Offer Split Test with 39% Uplift

It’s easy to think that all you need to do is put a good offer in the subject line and bingo you’ll get the best open rate.

But could how you talk about the offer make a difference?

In new subject line split tests run by Zopim marketing manager Abhiroop Basu, he split tested alternative offer messaging subject lines and found open rate uplifts of up to 39%. He’s shared all the results and let me include them here.

The list size was 80,000 and test cells of 3,300 where used for testing of five different subject lines. Results were validated using this split test significance calculator.

Before getting down to subject lines and results let me quickly explain the context.

Zopim is an online live chat solution with a free plan and monthly paid plans with more features. The objective of the email was to convert free users into paying users, with an incentive of a first month free on the paid plan.

Here’s a copy of the creative so you can fully understand the offer. The alignment of subject line to email body content is important to ensure you not only get the open but follow through with the click and conversion.

So let’s get down to the subject lines and test results.

The winning subject line (E) puts the offer message early in the subject line and makes it feel personal and exclusive by using the phrase ‘you’ve earned’.

The second best subject line also brings in a similar concept that the offer is a thank you gift. But this time placing the actual offer at the end, this means the offer could be truncated and not seen on some smartphones and email clients. It also requires the customer to read the whole subject line to get the message. Positioning of the offer at the end is very likely to be the main reason that E outperformed D by over 10%.

Subject line C also places the offer at the end. Comparing results with D imply ‘complimentary’ is a weaker word than free.

Subject line B was very weak compared to the winner and there are two potential reasons for this. Firstly the offer feels less exclusive as it doesn’t relate to being a current user and secondly the offer has more clarity. The subject line spells out that the free upgrade is a limited offer of free for the first month only.

I’d expect this second point to be the main factor. Before introducing the full details of the offer more selling is needed. So the subject line has to introduce the offer and the creative then support the offer value before explaining all the offer details.

When evaluating subject line tests I’d always recommend looking at the click and click to open rates as well as pure open rates. As a subject line that over-sells the offer can get high opens but fail to result in higher clicks. In most cases the objective is to get the click and not the open, so I favour subject lines that result in highest click rate not highest open rate.

Finally the weakest subject line, has initial focus on the new mobile optimized chat widget with the implication that the free upgrade is solely for this. The subject line likely turned off the customers who weren’t interested in mobile optimized solution and they didn’t get as far as finding out that this was not the only benefit on offer.

As is often the case, great tests like these answer some questions and raise more, helping guide further hypotheses for future tests. My takeaways from these tests are:

• Front load the subject line with the key points of the offer.
• Do enough to get the open and leave the email to sell the click. The subject line should be enough to whet their appetite and hook the customer only.
• Avoid a subject line that could reduce the size of targeted interested audience. Also be careful of the opposite, trying to target everyone, as this results in hooking nobody.
• Using free in the subject line is fine.

Commonly cited best practice is don’t use free in the subject line due to risk of classification as spam and corresponding drop in results. As these tests show, using free is not a problem. Not using free is one of the 7 subject line myths.

Abhiroop also noted that including the brand in the subject line was important and in separate tests a difference of 2.5% was seen by including the brand. This might well be because Zopim don’t tend to use their brand in the from name.

In addition to subject lines, Abhiroop tested calls to action, design and content. You can read more about these tests here, though with these tests because of the small sample sizes none of the differences had statistical significance.

Running split tests and getting no difference is quite common, when this happens read this advice on what to do when you have split test results with no winner. You can also avoid having no winner by adding an extra ingredient to you split test strategy.

As an aside I’ve tried out the Zopim solution. It’s amazingly easy to install and start using, if you’ve not tried live chat you might want to take a look at Zopim.