The strategy behind this incredible success involved something the email team called “Frankenstein testing”. I was fortunate enough to catch-up with Toby and he shared details of his strategy.

Before getting to the Frankenstein’s let me first give the bigger picture. Three things that were key to success:

• List size – the campaign did have a big quality database to leverage
• Email frequency – OK, so a lot of email was sent
• Test and optimization – up to x6 more revenue was produced per campaign by testing

I’m going to share with you their approach to the test and optimization piece of the puzzle. In Toby’s own words “we did as much testing as humanly possible”.

The fundraising emails were all about getting donations and so copy had to do most of the work persuading donors to open their wallets. The process used is as follows:

• A team of 18 writers created 4 to 6 different body copy messages for every campaign.
• Each body copy variation was sent out to test audiences with three different subject line choices
• Large test cells of 50,000 contacts were used to determine winners quickly
• The most important evaluation metric was simply the number of donations.
• The test cells were usually sent around 7am and a winner picked about 60 to 90 minutes later
• The winner went to the remaining database, but some segments received slight variations on the winner depending on their donation history.

You may be wondering how 18 writers could be afforded. Simple, the uplifts seen more than paid for them. On the ‘I will be outspent campaign’ of June 2012 the winning campaign raised $2.5m and a projected $2.1m more than the worst performing cell. That pays for the copy writers and the 10 analytics engineers checking over the results.

Occasionally although there was a winner, none of the emails performed to the level expected and in these cases the whole send was canned. The mentality was not one of send unconditionally.

Now let’s return to Frankenstein. This is the idea of taking the best bits of messages from different test cells to make a super email with the hope of creating monstrous results (cough, sorry).

In some cases a generic subject line was used to allow an evaluation of the body copy alone, such as use of the subject line ‘Hey’. This could be matched and make sense with any other body message.

A Frankenstein test might then take body copy that gave good results and match with a subject line that gave good results, expecting to see the two together being a best combination. Obviously this could only be done when the subject line was not so specific to the body that it would not have sense.

What was interesting was that the Frankenstein’s’ mostly didn’t turn out to be winners. Sometimes yes, but mostly not.

The learning was that the combination of subject line and body copy was found to be important.

The subject line impact is much more than just getting the open. How the subject line frames the subscribers thoughts and how they perceive the whole message as a result of the subject line plays a big role.

Subject lines must be optimized for the action and not just the open.

For completeness let me point out it wasn’t just subject lines and body copy that was tested, other items tested included; calls to action, donation amounts, text formatting and emphasis.

Acknowledgement: My sincere thanks to Toby for taking the time to share this information with me.

Around the internet there was a lot of fooling going on yesterday, on blogs, websites and Google Nose beta on their home search page. A few brands also did a little fooling in their email campaigns.

Here’s a couple of example email creatives, from Boden and Snapfish (HP).

Both of the fool emails sensibly stuck to a fool based around extension of their own product line. The Snapfish email introduced scratch and sniff photo printing whilst Boden focus on the new Man Skirt.

The Snapfish email included an animated GIF to show the smell wafting off of the picture, animating the Sniff and the Smell.

 Did you see any April fool brand email designs that made you giggle?

• Germany
• Germany
• Germany

I recently enjoyed some great German (yup) stand-up comedy from Henning Wehn on his ‘Henning Knows Bestest’ tour.

As we walked into the venue we were handed a post card to enter the competition. Of course I studied the entry post card more carefully than any normal person – with the kind of zealous attention to detail you’d expect from a German in fact. As well as the tricky competition question above, it asked for name, email address and postal address.

Even the Terms & Conditions followed Henning’s humour theme stating “Prizes administered by Henning’s UK agents – surprisingly reliable considering all their staff are English”.

But there was something missing, can you spot it?

Here’s a clue, its one of these

• No marketing permission collected
• No marketing permission collected
• No marketing permission collected

So the email addresses being collected can’t be used. Verflixt.

Quite a shame as the monthly Henning newsletter is a good bit of entertainment and anyone attending a gig would happily have that and information about future gigs arrive in their inbox. Sign up here if you like German humour or want to see if it really exists.

Here’s the kicker, Henning starts the gig by explaining he used to be in marketing. Guess we know now why “he used to be”.

Do collected email addresses at every customer touch point. But don’t forget the law, you have to get marketing permission to use the address and provide a way to opt-out at time of collection.

How can you know if you are engaging customers if you aren’t measuring them?

In the early years of email marketing with a monthly send campaign metrics did an adequate job. However the situation has become more complex. Frequency has been increased to many times a month for some brands, segmentation is used so that on each send different portions of the customer base is contacted and on top of that triggered email programmes are overlaid.

With all those emails flying around classic campaign metrics fail to capture this complexity.

As an example consider these four campaigns sent over a two week period. Due to segmentation each send is of a different size.

Now ask the question how many customers were engaged in this period?

The metrics can’t tell us. What if some customers are in just one segment and others in multiple segments? A customer may have clicked on zero, one, two, three or four campaigns. Are those 1,230 clicks from the same 1,230 customers or is every click from a different customer? We simply don’t know how many unique customers were engaged and by how much.

Now consider the customers who are in the database that were sent nothing, the metrics don’t reflect we didn’t even talk to those customers.

I’ve been working with three other DMA Email council hub members on what is needed to measure customers and engagement. You may download and read the resulting discussion paper we’ve authored.

Whilst a big step forwards this is a starting point, the intention of the authors is to further verify and improve this, however one thing is clear, we won’t be changing the view back to campaign metrics as a way of determining customer engagement.

Until next time, keep pushing the envelope.

The amount of subject line talk has meant many things are passed off as best practice but in fact are poor advice

Myth #1 Don’t use CAPITALs in subject lines

I believe this myth stems from the idea that capitals are shouting and its rude to shout, though another reason is that capitals are harder to read, the human eye can read lowercase faster. In many thousands of subject lines I reviewed I found only one in which the whole subject line was uppercase. In contrast the use of a few capitalized words in the subject line is relatively common.

Using capitals does help standout of the subject line and allows emphasis to be placed on key words. About once a month Miss Selfridge use a subject line that begins:

“PETITES EXCLUSIVE -“

The rest of the subject line depends on their offers that month. As well as getting standout the frequent (but not too frequent) use of this capitalisation trains customers to know, recognise and expect it.

Over the Jubilee Ann Summers ran,

“NeVer MiNd tHE MoNaRcHs?“

which certainly stood out and played on the Sex Pistols album of almost the same name. Whilst the alternating upper and lower case make it initially hard to read, the standout buys the extra few seconds of the consumer attention needed to read it.

Myth #2 Subject line name personalisation makes the email look spam-like

The concern is not that the email gets filtered as spam but that it is tacky, spam like and tarnishes a brand. Marketing opinions on this are often strong both for and against.

The most common personalisation is to include the first name at the start of the subject line, for example…

“Sarah, Weekend Super Saver 15% off”.

In analysis it’s very rare to see a first name included anywhere but at the start of the subject line.

In tests including a first name often gives an increased response rate. Better subject line personalisation goes beyond just the name and tailors the whole subject line message to the person or segment.

Real world examples are, Travelodge and Cloggs.co.uk who start every subject line with the first name whilst Disney includes it on about one in four subject lines. Since using the exact same approach on every subject line creates boredom and customers can tune out from it, the Disney approach of varying the subject line style makes more sense. If you do see a split test uplift with use of first name, do re-test regularly to check its effectiveness is not lost.

Myth #3 Don’t use spam trigger words like Free in the subject line

This is probably one of the most persistent old wives tales when it comes to subject lines. That use of “free” gets you filtered to junk. Many years ago there was some truth in this, as at that time email was largely filtered based on content. The biggest spam filtering factor now is reputation, which itself is heavily based on spam complaints.

Content is now only likely to get you filtered to junk if your reputation is weak.

Go ahead and use free and other sales trigger words, even put free in upper case too. Keep an eye on the inbox placement and if you see a problem then cut back on the sales words and resolve your underlying reputation issue. Once resolved you can restore use of these words.

Myth #4 Judge subject lines by the open rate

If a subject line creates lots of opens but few clicks and conversions just how good was it really? For most commercial mailings open rates are a weak measure of marketing objective. Click, post click and conversion measures are better metrics, even when it comes to deciding on subject lines.

It’s not uncommon to see a lower open rate subject line generate higher click rates. This happens when the lower open rate subject line is a better qualifier. Fewer people may open; however those that did were the right people and were more likely to click, leading to high click rates overall.

Myth #5 Keep subject lines short

I’ve nothing against short subject lines. The problem is with short subject lines at the expense of meaning and clarity. If the subject line is so short the reader doesn’t know what the email is about then it loses effectiveness.

Take these two short lines from Ryanair, “Book Today” and “£5 Seat Sale”. The second one is better as it says something about the email content. In this example the £5 offer was limited to midnight on the following Monday (48 hours from when the email was sent). Given urgency is a strong driver it might have been worth using a longer subject line and including the cut-off date. As a further example, had the offer been around particular destinations then by adding those it would better qualify the openers.

By contrast this 154 character example from Dorothy Perkins “Floral prints, peplum shapes and flattering silhouettes – say hello to Zen Garden! It’s battle of the denim with DP vs. Cosmo, plus 30% off all jewellery”

Determine your message and then aim to say it succinctly. Thus concentrate on getting your message communicated not the number of characters.

Myth #6 The only point of a subject line is to get an open

Many customers read your subject line but don’t open your email. The subject line message still has impact even when an open doesn’t follow. It leaves an impression with the consumer.

Your brand values, product range and more can be communicated through the subject line, even to non-openers.

Myth #7 I know what works

Whilst guiding principles can be used to write good subject lines, we are not the best judges of the best subject lines. Customers are the best and split testing is both an easy and effective way of understanding just what customers think.

Split testing subject lines is one of the easiest optimisation wins. The win is not just to get an uplift on the current campaign but through analysis of subject line differences creating theories to guide future campaign subject lines. This  avoids re-learning on every campaign and enables on-going refinement.

Conclusion

Fundamentally, subject lines are attention grabbers that need to resonate with the reader. Thus the best subject lines are written when there is a clear sense as to exactly who the reader is, what they want, how you express what’s in it for them and ensuring the subject line is well aligned to the content of the email.

Footnote: I first published this article on Smart Insights

Spam filtering works like blowing up a balloon,  if it bursts, the email is spam. Many individual issues contribute a puff of air into the balloon.

Some, such as spam complaints, give a bigger puff of air than others. Once the amount of air reaches a critical point, the balloon bursts and the email goes to spam rather than the inbox.

Just as for a real balloon, no one particular puff is responsible for bursting the balloon and no single puff will cause the balloon to burst, it is the sum of all puffs that’s important.

Seven things that blow up your balloon

The key items that cause a puff air into your balloon are, in order of importance are:

1. Data source and collection practice
2. Poor bounce management
3. Poor complaint feedback loop management
4. Content issues
5. Poor data management
6. Infrequent emailing
7. Lack of authentication (SPF, DKIM)

This is a list of things that most likely cause problems, not a list of all factors that go into spam filtering.

The number one point is data source and data collection practice. This is because how the recipient sees you and their state of relationship with you has a big impact on their behaviour and whether they hit the spam button or read your email.

Consider how customers got onto your list. Should someone provide their email address for, say, a quote, and not be clear they will also get marketing too, it will cause spam complaints. Purchased data will be much worse, not only causing high complaints, but high bounce rates and spam traps.

Data over content

So the first place to look when delivery is a challenge is your data, not your content or sending infrastructure.

Consider these important data questions:

• Where did your data come from? From organic sign ups, competitions, purchases?
• How was expectation set in terms of how the address would be used and what would be sent?
• Has any old data been reintroduced?
• Has data always been well managed?
• Has unsubscribed data been accidentally re-introduced?
• Has the unsubscribe process stopped working?
• Has purchased data been used at any point?
• Are hard and soft bounces being removed?

Keeping a record of data sources is not only good for marketing performance, but also for deliverability. If you need to purge data to improve deliverability, then removing the less active data from the most risky sources first is a good strategy.

If you have squeaky clean data and emails are expected by recipients, then your balloons should not burst. The other factors such as content can play a part, but typically only if your data is not in perfect order.

There’s advice aplenty for creating a better user experience on mobile devices and use of responsive design in email templates. But just how important is it? Should this be your number one priority? The data suggests  not.

The logic behind the need for mobile design is self-apparent. If users get a poor experience of your email on a mobile device then they are going to delete and ignore you.

If you’ve ever tried to use a website on a mobile device that’s intended for desktop use, then you’ll know just how frustrating it is and how quickly you give up.

Emails are somewhat simpler than websites; just scan read and maybe click on one or two things. Not the same sort of reading, form filling, navigating, multimedia interaction of a website.

Is the typical email today too bad for mobile use? Just how good does an email need to be for mobile use and just how far short are emails today? This was worrying me and I got chatting with Steve Henderson from Communicator Corp. We started developing some theories and Steve went to work looking for evidence and data to verify or reject these ideas.

The first question to answer was that whilst 43% of opens are mobile, as reported last December by Litmus, what is the spread like? Are all brands seeing something very close to this? Are some brands so unaffected they don’t need to consider mobile?

Mining the data from 377 million emails, 339 brands and 12,000+ campaigns we charted the number of campaigns against the ratio of opens on a mobile device to all opens for each campaign.

The chart below shows the result of this first step in analysis.

For example, this chart shows that 14% of all campaigns analysed showed the ratio of opens on a mobile device to all opens was in the range 50% to 54%.

There’s a good spread with most campaigns in the range of 25% to 65% of opens happening on a mobile device. There are still campaigns at both ends, some with almost no mobile activity and some with almost only mobile activity. It’s possible your campaigns sit at one of the ends and not the middle. If you’re at the bottom end then mobile optimisation is crucial issue for you.

To answer the question of how important is mobile design we need look at open and click rates across this chart. If emails not optimised for mobiles are providing a significantly poorer user experience then I would expect to see the average opens and clicks lower for those campaigns in which a high number of opens on mobile devices occur.

The next chart is the same as the previous but with average open and click rates for each of the vertical bars added.

Surprise! The open and click rates don’t vary much compared across the chart and certainly no downward trend that would show mobile email is an issue for consumers. The open rate actually shows an upward trend. This is most likely due to the iPhone loading images by default and therefore triggering open tracking by default. The click rate is a more honest measure and is almost flat.

There’s nothing here to say that open and click rates are in danger of dropping because of consumers moving to mobile devices. Actually, the average email works to an acceptable degree without modification. Smartphones do a good job of making email usable. Is responsive design just another bandwagon you don’t need?

The self-apparent conclusion that responsive design is a must goes unproven. I’ve discussed email mobile design with some of the most experienced responsive design gurus in the industry. They’re telling me in some cases they have seen good uplifts as a result of responsive design.

How can responsive design deliver an improved response with the evidence above showing it’s totally unnecessary? One reason could be because the method of measuring success of a responsive design has not been robust – with multiple factors changing, different offers, external factors, timing and much more.

There’s another reason too. Mobile first thinking and responsive design applies a much stronger focus to key campaign objectives and drops all the clutter in the email. Screen space is a premium so only the important content, links and calls-to-action are included. Even more the calls to action and headings are made even bolder.

A responsive design is just a new design and any uplift seen is not because of design for mobile but simply designing a better performing template.

Right now my conclusion is simplify your template, make it focused and get rid of the clutter. It will just work better, regardless of desktop or mobile.

Finally, a note about the bars at either end of the chart: the very low end is showing reduced opens and clicks and the very high end increased. There are two reasons for the outliers performing like this.

• In the sample data there are not many campaigns at either end, meaning individual brands are skewing the results.

• At the low end a B2B brand with informational emails that don’t require open or a click to be useful has pulled the average down and at the high end a brand is sending some mobile specific emails to only customers who open on mobiles, pulling the average up.

I was recently speaking at the big Email Evolution conference in Miami and discussed in one session the pros and cons of responsive design for email with brands who had created mobile websites and looked at email performance on mobile. They agreed with the general view that responsive design is not always necessary and in the session other people shared their stories of sometimes it helped and sometimes it didn’t.

Footnote:

The following assumptions about the data used this analysis have been verified as reasonable:

• The customer base of each brand varies in the number of mobile users

• No one brand sent sufficiently large number of campaigns compared to the other brands to skew the results

• The brands included have typically the same amount of mobile optimisation, it is not the case that brands with more mobile users are typically more optimised for mobile

• Different campaigns are not generally sent to a mobile/non-mobile customer or targeted on mobile activity

• Mobile opens in the analysis includes smartphones and tablet devices

Sample sizes are all about statistics; fear not, you’ll get the information you need here without a formula in sight.

Let me start with an example of an A/B test using a sample size of 100. With a result of test cell A giving a 5% click rate and test cell B 6% then logically B is the winner. The relative uplift of B on A is 20%. That is a 1% increase on 5%, giving increase 1/5 = 20%.

Sounds great? With the sample size of 100, test cell A 5% click rate means 5 people clicked and in cell B 6 people clicked. Just one person difference. If just one person had gone the other way there would have been no difference between cell A and cell B, no winner, no 20% increase.

The result is termed not statistically significant. This means that the difference between the test cells is due to random variation and not because of a true difference in the effectiveness of getting the click.

Clearly jumping to a conclusion based on one person difference is totally unreliable. Of course a larger test cell is needed. But how much larger, should it be? 200, 300, 4000?

Help is on hand to allow you to decide your test cell sample size by using this simple table.

To use the table you need to know just two things:

• What is your normal response rate?
• What is the smallest difference you want to measure?

Your normal response rate
This is simply the normal click through rate you expect. If this is the first time for the type of campaign being tested then make an educated guess based on your other campaigns. Guessing on the low side will play safe with sample size.

Smallest difference to measure
The size of the difference in response between test cells affects how big the test cells need to be. You must trade off test cell size against the smallest difference in response that you want to know is a true difference, a difference that is statistically significant.

To help answer this question think about the level of bottom line improvement worth testing for and the cost of doing the testing. Typically its not worth trying to measure less than a 10% difference, for example, less than an increase of 5% to 5.5% is not interesting.

As data is a precious resource using the smallest possible test cell means more test cells and more tests.   Running more tests searching for a 10% or 20% increase is better than one large test cell that allows a 1% increase to be measured with statistical significance.

Using the table
Let’s say our normal click rate is 10%. In the test we want to measure if the click rate changes by 20% or more.  That is if it increases from 10% to 12% or more. First look down the left hand table column and find 10%, then look across to the column for 20%. This gives the answer that the sample size needed for each test cell is 2000. With this sample size you can be confident that any click rate change of 20% or more is a statistically significant result and not just randomness.

I’ve used click rate as the optimisation response metric throughout this post. The table and the same concept applies whether for open rate, click to open rate or conversion rate. You can plug those alternatives into the same table. Just remember if you use a click to open rate your sample size will be smaller. The sample in this case is the number of people who opened, not the number of people to whom you delivered.

Which metric you should use depends on the metric that best represents your marketing objective. Hint, that’s unlikely to be your open rate.

Should there be any students of statistics reading, you may wish to know the table above is based on a confidence level of 95%.

I hope that you are now better equipped and know better next time you hear someone saying ‘just use 10% of your list to test’.

There have been a couple of heated debates I’ve been part of in the email community just recently, one about open rates and another about subject line length. In this post I’m reflecting on the open rate debate. I’ve added an update to the original post showing more proof that open rate fixation can be misleading.

It may surprise you that the community is very split on the value of the open rate metric, seemingly, a cornerstone metric since the dawn of email marketing. It’s a metric cited in every email marketing book and on every mail marketing course… So why is its value in doubt? Ken Magill reported on some of the open rate debate.

Let me summarise some of the reasons why open rates are not important.

• Opens are almost never the objective of a campaign. Possible exceptions might be in publishing and list hygiene, when identifying active addresses to solve deliverability problems.
• Higher open rates do not always mean higher click and conversion rates. Open and click/conversion can and do go in opposite directions. Why bother with open rate when you can focus on click or conversion?
• Open rates can be misleading. Through focusing on driving higher open rates this can potentially leave you with less business.

I’ve recently published some research for Emailvision reviewing whether open rates are a good predictor of success in email marketing across 50 million retail emails from 196 campaigns.  The headline finding is…

Open rates wrongly predict success 53% of the time

As you might expect, there is a much closer correlation between clickthrough rate and open rate. This graph from the research shows that the open rate is relatively unchanging; it typically varies by just 12%. So if the open rate average is 20% that means it varies up to 22.4%. However conversion rates vary hugely in comparison, by some 70%. If average conversion rate is 0.2% then it varies up to 0.34%. To understand campaign performance the conversion rate is a must.

Why can a higher open rate mislead and deliver less business? Take an example, I talked to someone recently who was delighted they had increased their open and click rates. I asked how. The answer was they removed people who hadn’t opened recently. I asked if they had conversion and revenue figures to compare. I wasn’t surprised when I didn’t get an answer…

The five different cases below illustrate many of the problems with open rates.

Case 1 – List hygiene and inactives

List size 10,000
Number of people who open 2,000
Open rate 20%

A common percentage of inactives is 40% of the base. Let’s remove those 4000 people from campaigns.

List size 6,000
Number of people who open 2,000
Open rate 33%

That’s a 50%+ increase in open rate. Will it have resulted in 50% more revenue? Absolutely not. In fact most likely the revenue will decrease, because every month a percentage of the apparently inactive customers do come back again.

The open rate metric and increase in this case does not represent business success.

Case 2 – Monthly reporting and averages

Campaign open rates are sometimes averaged over the month for reporting. For example these three campaigns:

Campaign A  60%
Campaign B  20%
Campaign C  10%

This gives the monthly open rate, calculated as the average of open rates, of 30% = (60+20+10)/3.

Let’s add real numbers to the above figures

Total emails sent 10,000 with total opens 1600, giving an open rate over all campaigns of 16%. The commonly used average of opens, at 30%, is almost double the real value.

Case 3 – Same rates, but one campaign is better

Two campaigns with identical open rates. Which campaign was more successful? Ok, so you’re getting wise and want to know the send size before you decide? That’s fair, here are more campaign metrics.

They still look the same. However, campaign Y is more successful. Because the average order value for campaign Y was double that of Campaign X.

Case 4 – The impact of frequency variation and on open and click rate

Customer segment A receives a campaign with open rate 20% and click rate 5%.

Customer segment B receive two campaigns in the same period and because of email increased marketing pressure the open rate drops to 15% and click rate to 4%.

Which was more successful?

If the list size is 10,000, then the number of opens for segment A was 2000 and for segment B was 3000 across the two campaigns. Making campaigns to segment B more successful, although the campaign open rate is lower.

Case 5 – Subject line A/B split tests

Surely the open rate is a useful measure of the subject line quality?

Well, not really here either. Take the case where one subject line gets lower opens and higher clicks than the other. The better subject line is the one with higher clicks and lower open rate. Fewer people may have opened but the ones that did open were the right ones, the right target for the offer and call to action.

It would be even better to evaluate the subject line on conversions, however test cell sample sizes don’t normally permit that.

If not open rate then what?

To sum it up, the better metrics to focus on are:

• Total revenue from email
• Revenue per 1000 emails sent (customers) per campaign
• Email revenue per customer per month
• Average customer order value
• Conversion rate
• List size and growth
• Percentage of email base purchased in the last month
• Percentage of email base clicked/opened in the last month

Notice how these metrics consider money and customers not campaigns. Measure campaigns to understand what campaigning strategies work and measure customers too, as customers are more fundamental to business health.

What other metrics, whether new ones or old ones should have time called on them?

Acknowledgement: I originally posted this article on SmartInsights