## June 01, 2022

### Show Me: An A/B Test and the Organic Percentage

Some of the email I receive outlines concerns about the Organic Percentage. The emails look like this:

• "We just executed a month-long email A/B test. 50,000 customers received the normal stream of email campaigns (5 per week), the other 50,000 customers received no email campaigns. We did this for a month. At the end of the month the group receiving campaigns spent \$10.00 per average. The group not receiving email campaigns spent \$8.50 per average. Our email service provider says we generated \$0.12 per campaign during this time, netting \$0.12*20 = \$2.40. What is our organic percentage?
Technically you have two different "organic percentages" to think about. The first one is the \$8.50 / \$10.00 = 85% metric. If you don't send email marketing campaigns, the customer will spend 85% of what they'd normally spend.

Then you have your "incremental rate" ... not a true organic percentage but it is a cousin of the organic percentage. Your email service provider says you generated \$2.40 from email marketing. Your A/B test shows that you generated \$10.00 - \$8.50 = \$1.50 from email marketing. Your "incremental rate" is \$1.50 / \$2.40 = 62.5%. For every dollar your email service provider says you are generating from email marketing, you are actually generating \$0.625 of true incremental value with \$0.375 being cannibalized from other activities.

In other words, you have to measure two different things to get a picture of what is happening. Measure your organic percentage to understand how much the customer spends independent of a marketing channel. Measure your incremental rate to understand how much your marketing vendor is overstating your results.