Use these two calculators when split testing to check if your email campaign results have statistical significance and to decide sample sizes when planning new tests.
Results check for statistical significance
To check your A/B split test results are valid just enter the metrics into this calculator.
For example if both test cells used a sample size of 5000 customers and the click rate on email A was 6.7% and B 8.9%, then entering those values shows the uplift has statistical significance of 99%.
You can enter open rate, click rate, open to click or conversion rate – in fact enter whatever rate you’ve decided as the best metric to evaluate success.
There is no statistical difference between your A and B samples. Either test with bigger test cell sample sizes or change what you are testing. The % uplift has a weak 80% level of statistical significance. Consider whether you want believe there is a real difference or retest. Looks promising, your uplift of % has a statistical significance of 90%. Good work, your uplift is % with a good statistical significance of 95%. Congratulations, your uplift is % and you can be confident, as the statistical significance achieved is 99%
Is your test strategy holding you back from bigger uplifts? Find out if your results are limited by this common split test misconception.
Email split test sample size planning
Use this calculator when planning a test to decide on the test cell sample size needed.
If you want to find out which test treatment gives the best click rate, then enter your typical campaign click rate and the uplift for which you want to know the sample size will be large enough to give the result statistical significance. For example, if your typical campaign click rate is 8% and you want to be certain that a 10% or more uplift in your test is not random variation, the sample size must be 9235.
The sample sizes calculated above are based on 95% significance.
Looking for ideas and strategies to improve results? Get our free email marketing tips.