A great example of an email split test, 300 subject lines suggested by readers of Marketing Experiments, reduced down to ten and a split tested.
The email split test results table taken from the original post is below but before you check out the original post read on here for a quick lesson in split testing strategy not mentioned in the original post and rarely talked about.
The top three subject lines in the table gave identical performance within bounds of statistical significance. Had only those three been tested there would have been no winner, no uplift and no lesson to apply in future campaigns.
The next five subject lines, lines 4 to 8 also shared identical results. No winner amongst these either. Had the test been just these five lines there would again have been no lesson or uplift achieved.
This illustrates an important key to success in split testing, you need skill and a sufficient quantity of tests to give lady luck a chance. The key ingredient missing from many test plans is simply doing enough testing. Don’t give up with split testing too soon, don’t try one or two tests and then stop because you didn’t hit a home run.
As a rule of thumb expect out of ten test treatments for one to make a big shift to the needle, 3 provide some learning and improvement, 3 no winner and 3 under perform. On this basis running one test a month means one big learning per year.
We incorporate split testing as part of strategy with all our clients – except where there is insufficient data to make testing feasible.
As an example, in a recent email split test we ran 11 treatments. Of those 4 produced uplifts, 2 had the same performance as the control and 4 under performed. Two of the 4 winners gave over 30% improvement in click through rate. We couldn’t have got this result without incorporating quantity in our plan.
If you are running A/B subject line tests then be prepared for several tests showing no performance improvement. Don’t get despondent but keep persevering.
Don’t forget that your test cells must be large enough for validity, you won’t get a valid result by running 10 split tests on a list of 5000. Use this split test calculator to work out test cell sizes and check your results.
Also take a look at this advice if you get identical split test results. It provides more tips as to what to do next.