Results and Lessons from 9 Subject Line split tests

A/B subject line split testsAt the recent Email Insider summit the consensus was the number one element to split test in your emails is the subject line. So I’ve rounded up 9 subject line split tests.

For each of the nine tests you’ll find the subject lines tested, the results & winner and what you can learn for your campaigns.

1. Travel industry

A: Time is almost up. Click below by 30th for your chance to own an iPad

B: Are we still welcome in your inbox? iPad prize for grabs until 30th November.

C: Are we still welcome in your inbox? Please reply by 30th November

The Result

Subject line C outperformed A with 143% increase in positive response. Subject line B was a close second.

The lesson

The campaigns were targeted to inactive travel customers who had not opened an email in over 9 months. It was the last email in a sequence of three.

I’ve often found with inactive audiences a strong sell in the subject line is a turn off. Possibly since the majority of B2C campaigns have a strong sell it’s necessary to do something different to get attention from inactives.

Also with inactive segments I’ve generally found questions work well. So consider testing a question based approach for different segments of your audience.

2. Tumbleweed Tiny House Company

A: It’s FREE. All the tiny houses on our site and more

B: It features all the houses on our website plus more…

The Result

Subject line A won with a 26% higher open rate.

The Lesson

Deliverability concerns when using free in the subject line is one of seven subject line myths that continue to circulate the internet. So it’s great to see this test by Tumbleweed Tiny House Company, shared by Aweber, taking this head on and showing using free is not a problem for deliverability.

Free continues to be a special price point that has a high probability of making an impact and worthy of a test.

The click rate for this test wasn’t shared and I would have wanted to evaluate that too. It’s not unusual for a campaign with higher open rates to yield lower clicks, in fact open rates wrongly predict success 53% of the time.

3. Expedia CruiseShipCenters

A: Set Sail and Save

B: Dave, Set Sail and Save

The Result

Subject line B had 17% higher open and 19% higher click rate.

The Lesson

A classic personalisation test by Expedia CruiseShipCenters in a case study from Lyris. Using a first name, last name or even both commonly produces an uplift – though as with any tactic it’s not universal.

The lesson is to test personalisation in the subject line. This doesn’t need to someone’s name; it could be the name of an event they attended or their preferred brand.

When a customer really feels the subject line speaks to them personally it grabs their attention. Try testing names and use of dynamic content to deliver an even stronger personalised subject line.

4. BrightTALK

A: Live tomorrow: Next Generation Desktop Summit

B: Live today: Next Generation Desktop Summit

The Result

When sent to an in-house list subject line A gave an increase in opens of 111.7% and in clicks of 153%, however, it was B that gave more event registrations.

The Lesson

The subject line that generated the most interest didn’t generate the most conversions. In further details of the test BrightTALK shared they also ran the same test to an acquired list. For this less engaged list the opposite was seen, with A generating the most conversions. Proving that different audiences can and do respond differently. Just because someone else had a winning subject line it may not be a winner for you.

Also the test shows the value with measuring and evaluating tests right through the funnel to conversion. But remember measuring changes in conversion events requires much larger test cell sizes for valid email split test results. Cell sizes that are fine to measure open and click rate changes may not be large enough for conversion rate change measurement.

5. United Agencies West Insurance group

A: $20 Million Dollar Homes Burned To The Ground

B: Protect Your Home from Wild Fires – Brush Fire Season is Coming

The Result

Subject line A won increasing leads by 65%

The Lesson

Subject line A invokes emotion and grabs attention rather than the simple sales line in subject line B, in this test reported by Macquarie IT.

The winning subject line reads like a newspaper headline, has elements of fear and intrigue that makes the reader go further. I’ve also seen attention grabbing subject lines produce higher open rates but fail to produce higher click rates. This can happen because the people who open are less qualified to the offer. Another good reason to ensure you test and to determine winners by clicks or conversions rather than open rates.

A good lesson from this test is to test totally different approaches to the subject line rather than minor tweaks. Try testing; urgency, clarity, intrigue, emotion and questioning subject lines.

6. UBM Aviation

A: The Airline E&M China & East Asia conference returns to HKG next March

B: Invitation to Hong Kong – Airline E&M China & East Asia Conference

The Result

Subject line A, whilst generating no difference in open rate gave a higher click through rate.

The Lesson

The impact of the subject line is deeper than just opens. In these test results published by Adestra, whilst the subject line appeared to make no statistical difference the click rate did show a strong statistically valid increase. I recommend testing subject lines designed for action not just for the open.

7. dotMailer

A: 10 email templates to send your festive invitations & surveys out

B: ✭10 email templates to send your festive invitations & surveys out ✭

The Result

Over multiple tests using symbols like subject line B gave an average click increase of 15%

The Lesson

dotMailer detailed how they tested adding a star to subject lines over multiple campaigns in this test. Two important lessons here are to test multiple times to check results are repeatable and to be consistent in the test. They didn’t change anything except adding the symbol and they tested consistently with the same symbol.

8. Litmus

A: Mobile opens take the lead: new market share stats

B: Mobile opens take the lead with 80% increase over 6 months

The Result

Both performed identically, no winner.

The Lesson

Litmus ran this test to see what uplift could be gain from using numbers in the subject line. Because numbers have strong believability, provide evidence and are quick to understand they often drive better results. It’s a great test.

So why didn’t numbers provide an uplift? Perhaps because the way in which the numbers were used. The numbers didn’t incentivise someone to read further. The subject line is a simple statement that gave the story in one line, leaving no reason to go further. A teasing subject line with numbers might have worked better, such as “Latest mobile email engagement metrics from 428 million emails over the last 6 months”.

Getting no difference in results and no winner is quite common in testing. Almost all published test results only talk about the tests that gave remarkable results and don’t mention this one ingredient you need in your test strategy. Should you not get a winner in your test then follow this advice when email split test results are identical.

9. Internet Marketer

A: 1 simple secret to skyrocket the quality of everything you do

B: How I ‘hack’ excellence with 1 simple trick

The Result

Subject line B had a 12% higher click rate – but read on below.

The Lesson

The lesson in this test is probably not the one you’re expecting. After checking the results (see point 9 here) the sample size (2700) and uplift are not sufficient to be certain of a winner, there is weak statistical significance of 80%. The click difference could be just random variation and it’s misleading to draw conclusions and build future campaigns on any perceived insight.

There are many split test stories around the internet that lack the rigour needed for solid and continuous optimisation.

The simple lesson is check your results, such as by using our free online significance test calculator.

The final takeaway

The best use of other peoples test results it not to blindly copy their winner assuming it works for your audience and context, but rather to use the test for insight and inspiration in your own tests. I hope this article helps you with that.