Dear Catalog CEOs: Statistical Significance

Dear Catalog CEOs:

Statistical significance is a concept that few people understand ... even those who preach A/B testing!

In business, statistical significance is an arbitrary measurement.  It represents the minimum probability that one outcome will outperform another.

• Test Group Conversion Rate = 5.50%.
• Control Group Conversion Rate = 5.00%.
• p = 0.05.

This outcome is statistically significant at a 95% level.  If this test were to be replicated 100 times under identical conditions, 95 times the test group would outperform the control group, 5 times the test group would not outperform the control group.

There are several problems with this outcome, problems that most business leaders (and most statistical analysts) fail to grasp.

1. The test does not guarantee that the test group will outperform the control group by 10%.  In fact, this outcome will almost never be replicated again!  So many CEOs have a problem with this issue.  I once worked for a CEO who told me that he wanted a guarantee that one outcome would always perform at least 10% better than the control group.  Well, in order for that to happen, you'd need a difference in the test/control group of maybe 30% to 40%!
2. Statistical significance is arbitrary.  Math lovers pick a 95% level.  What if the outcome of a test suggests that the test group will outperform the control group at a 94% level?  Do you say that it is not statistically significant?  No, that's garbage!  Testing is all about odds.  In business, I like odds that are in my favor.  If one outcome is better than another at an 83% level, I have a 5 in 6 chance of success.  How often in business can you say that you'll be right 5 times in 6?
3. Technically, your test results are only valid for the conditions under which they are tested in.  If you run a test on a Monday in August, you may not be pleased with what happens when you roll out your test to the entire audience on a Thursday in November.  Most people miss the subtlety of this one ... test results are dependent upon the conditions that exist at the time of the test.

Cut your folks some slack, when it comes to testing!