≡ Menu

Why you must split test more than one change at a time

Optimise a pig into a swan?A common misconception with A/B split test optimisation is that you must only test one thing at a time. This is not always the case or even good advice. Let me explain with the help of a pig.

Imagine starting with a pig and trying to optimise it into a swan. There are some reasons why you cannot create a swan from a pig

  • It takes too long, consider how long it would take changing just one thing at a time to make a swan
  • The new animal can only be successful when many things have been changed together so changing just one thing won’t show success
  • There is not enough data to do the number of tests needed. Test cells need to be around 5,000+ in size and this limits how many tests can be conducted (see also how to pick test cells sizes)

The problem with the pig is that the whole concept is too different to the animal we want to create. We are simply starting in the wrong place to get to where we want to be.

Evolution and revolution

Testing approach can be divided into evolutionary and revolutionary.

  • Evolutionary is making small changes and testing one thing at a time to evolve an email and create a better version of the same concept
  • Revolutionary makes many changes and may change the whole basis of the concept

Both approaches are valuable and have their place. I often use evolutionary and revolutionary approaches in a test plan.

Use revolutionary testing for a totally new concept and if it looks promising further refine using an evolutionary approach.

If an email is fundamentally not the overall right approach then whilst optimising using an evolutionary approach will make gains, the bigger gain is missed.

The point is that your starting position makes a difference. It may be faster to make several changes, find an uplift and then test to understand which elements were important to the uplift.

What about multivariate testing?

Multivariate testing means you can test more than one thing at a time. But there are still limitations as to how much can be tested whilst being able to pinpoint an individual change as the reason for an uplift.

This means the revolution approach of making sweeping changes is still a valid method to use.

Finally, as a rule of thumb, if you want to see a big change in result, make a big change – though not randomly of course, with a plan and hypothesis behind changes. Yes, there are cases of changing a few words or colours that make a big difference, but generally bigger changes have bigger impact

Comments on this entry are closed.

Free Email Marketing

Strategy Tips & Advice

  • Actionable strategy advice
  • Quick win tactics proven to work
  • Actual results and case studies

We won't share your email address

x