What Does An Email Mail/Holdout Test Look Like?

More than twenty years ago at Nordstrom we personalized every email we sent, and we executed mail/holdout groups. We knew that email was more productive than what our open/click/conversion metrics showed - we knew it was worth the marketing investment to personalize all of it and then generate another 20% on top of something that was already wildly profitable and not getting full credit.

We did that back in 2002. That's twenty-three years ago. Think about that for a moment.

You're emailing customers 10x a week, so you could hold out a group of customers for three to fourteen days, learning how customers interact with all of your channels.

After a few weeks, you average demand/sales by attributed marketing channel.

This is a very common outcome in an weekly email mail/holdout test. The two numbers to focus on are bolded in blue above.

• The group that received emails all week spent $0.50 per customer. This is what you'd normally report across maybe 5-10 campaigns, in total.
• The group that did not receive emails all week spent $2.00 - $1.54 = $0.46 per customer of incremental volume.

In other words, 92% of what this company commonly reports as email marketing revenue is actually email marketing revenue ... the other 8% would happen anyway if email marketing campaigns were stopped.

It's common to work with clients where email marketing might account for 15% to 20% of annual volume. That's a lot of volume! In this case, we'd multiply 15% to 20% of annual volume by 92% based on the test result, which means that email marketing is truly responsible generating A FORTUNE!

I've been in meetings where we are able to show that a company with seven percent pre-tax profit would be unprofitable without email marketing. You should see the eyes on the Finance folks when that fact is shared. And yes, somebody then says they can't believe the science behind a mail/holdout tests though they sure seem to appreciate the math behind clinical trials.

Email marketers almost never get the credit they deserve. A small number of under-resourced individuals do all this work and then present the findings via opens / clicks / conversions, which are metrics that your average employee struggles to be passionate about. Profit? At worst case, your Finance team becomes interested.

I've been in the meetings where the email service provider tells my client not to execute tests ... "oh you don't want to do that!". Why not? Seriously. Why does a vendor get to tell you what you should and should not learn (I know, you're going to say why does a rogue consultant get to tell you what you should and should not learn).

Did you know that when I was VP of Database Marketing at Nordstrom, back in 2003, my testing budget for marketing channels was 3% of annual e-commerce net sales? In other words, if we were a $400,000,000 e-commerce channel, we were willing to forego $12,000,000 in annual sales to learn how our customers interacted with retail stores, our website, and our catalog. Hint - that's how we learned we didn't need to mail catalogs anymore.

You want to prove your value to your company and then secure resources to do the stuff you've always wanted to do that nobody will give you money to execute because the people who have the money don't align with opens / clicks / conversions.

I worked with a client a decade ago who had an email professional who personalized all of her campaigns and generated what appeared (via tests) to be a 50% increase in the productivity of her campaigns. She didn't take the logical step to convert her numbers to profit ... so when she left the room one of the Executives looked at the Executive next to me and said "WHAT A NERD".

That's what you are up against. That Executive likely earned $250,000 a year plus an 80% bonus that allowed him to purchase a new BMW every year if we wanted to do so ... all because this young woman was generating most of the profit at her company. And he had to belittle her ... she's the problem.

No. She's the reason the brand was profitable.

Execute an email marketing holdout test.