Show Me: An A/B Test and the Organic Percentage

Some of the email I receive outlines concerns about the Organic Percentage. The emails look like this:

• "We just executed a month-long email A/B test. 50,000 customers received the normal stream of email campaigns (5 per week), the other 50,000 customers received no email campaigns. We did this for a month. At the end of the month the group receiving campaigns spent $10.00 per average. The group not receiving email campaigns spent $8.50 per average. Our email service provider says we generated $0.12 per campaign during this time, netting $0.12*20 = $2.40. What is our organic percentage?

Technically you have two different "organic percentages" to think about. The first one is the $8.50 / $10.00 = 85% metric. If you don't send email marketing campaigns, the customer will spend 85% of what they'd normally spend.

Then you have your "incremental rate" ... not a true organic percentage but it is a cousin of the organic percentage. Your email service provider says you generated $2.40 from email marketing. Your A/B test shows that you generated $10.00 - $8.50 = $1.50 from email marketing. Your "incremental rate" is $1.50 / $2.40 = 62.5%. For every dollar your email service provider says you are generating from email marketing, you are actually generating $0.625 of true incremental value with $0.375 being cannibalized from other activities.

In other words, you have to measure two different things to get a picture of what is happening. Measure your organic percentage to understand how much the customer spends independent of a marketing channel. Measure your incremental rate to understand how much your marketing vendor is overstating your results.