Even better the clicks uplift was for an inactive segment and the winning subject line was itself being pitted, in an A/B split test, against a previously strong performer. So this was not a easy win against a poor subject line.
Re-engaging inactive customers is an important element of all email strategies. As this case study shows, the gap between good and great for inactive customers is wider than for active customers.
The details below include how it was done and a drill down into the performance difference between active and inactive customers.
Money Dashboard is a superb free online solution that keeps track of your spending across multiple bank and credit card accounts giving a true view of finances. It automatically aggregates and categorises spending to make money management effortless.
They have a large and active customer base. In common with all online services, they also have customers who registered but are not making full use of their service. Email is used to help engage those customers and convert them into active users.
The email strategy includes a carefully designed email sequence. Whilst each email has different content and individual objectives, most of the messaging is focused on highlighting the benefits of Money Dashboard to motivate customers into action and addressing objections that could otherwise stop them.
To be effective it requires an excellent understanding of the customer. Built into the strategy is the use of customer surveys to hear the customer voice. The survey insight is used to echo benefits in the messaging using the customers’ language and to guide what objections must be overcome.
Every individual campaign includes split tests and lessons taken from each campaign are rolled into the next. This particular campaign was an email to gain feedback through a survey.
In this case the campaign included an A/B subject line split test. Results tracked for both active and inactive customer segments. Inactive is defined as not opening or clicking an email in over 5 months and not a recently acquired user. Since recently acquired haven’t been given fair chance to click yet.
The two subject lines tested on the survey invitation email were:
- A: What’s your opinion here first name?
- B: Please put us out of our misery
The following table shows the uplift of the treatment (B) on the control (A) for both active and inactive segments:
| Open Uplift | Click Uplift | CTOR Uplift | |
| Inactive | 104.5% | 228.5% | 60.6% |
| Active | 30.5% | 103.3% | 55.9% |
All metrics reached a statistical significance of over 99%. Make sure when you test your sample sizes are big enough and use a free calculator to ensure results are statistically valid.
The control subject line incorporates the following concepts into its design:
- Appeals to the customer desire to be heard, people like their opinion to be asked and valued.
- By asking a question it creates an open loop of thought, motivating the open.
- Uses the customers’ first name, which in tests almost always improves response.
- It’s all about the customer and their views
Whereas the treatment takes a different approach:
- Engender an emotional response, seeking pity
- Use intrigue to create an open loop, raising questions such as what misery and why?
- The subject line is unusual, misery if not often used in subject lines.
The treatment is very us (brand) focused and not about the customer or what they might want. Best practice copy is to place customer and what’s in it for them central. Because of this we did not expect the treatment to win by such a large margin.
But the treatment doesn’t totally ignore all best practice, its leveraging emotion and intrigue. Both can be powerful. Nonetheless, it reminds us that best practice is just generally good guidance and not always the best performer. We went ahead and tested knowing we were breaking one best practice rule.
The fact that the treatment did outperform the control means the customers have sufficient affinity to Money Dashboard to have a sense of caring. It’s conceivable that this subject line on a customer base with lower brand affinity would not win.
It also shows in this customer base just because someone has not opened or clicked in over five months does not mean they are disinterested in the brand. In fact the inactive open rate was 22.7% of the active segment open rate. Given the active segment open rate is well above industry benchmark that’s pretty impressive for supposedly inactive. Your inactives may well be more active than you think!
The insight from the many survey responses and the subject line results are again rolled back into future campaigns. The right survey questions are invaluable in helping to create better campaign messages, if you don’t already use surveys it’s well worth doing.
As this test shows, it’s good to knowingly break a few best practice rules, subject lines that can trigger the right emotion will perform well and for your inactives it’s even more important to find a different angle to the norm.
Note 2nd August – I’m delighted to say this work has won the Gold award in the Which Test Won 2014 awards email and mobile category.