"I Just Don't Believe You"

That's how an Executive once responded when discussing the results of a mail/holdout test.

The results were really straightforward. Take an average of the twelve-month buyer file, select 50,000 customers randomly, then resample the group of 50,000 into two groups ... one receives the catalog, one does not receive it. In the six weeks the catalog was active, I measured total demand spent by the mailed group, I measured total demand spent by the control group.

• $6.00 spent by the mailed group.
• $4.00 spent by the control group.
• $6.00 - $4.00 = $2.00 of incremental demand.
• Organic Percentage = $4.00 / $6.00 = 67%.
• Profit = $2.00 * 0.40 - $0.90 = ($0.10).
• To The Twelve-Month Buyer File, On Average, The Mailing Was Unprofitable.

It was the last bullet-point that shook the Executive. The very thought that the heart and soul of his customer base was being managed in an unprofitable manner by ... the Executive himself ... the fact that the Executive was actually harming his business ... was unacceptable. 

So he told the room ... "I just don't believe you".

The room moved forward, harming the business.

If you are unwilling to believe the most simple level of scientific inquiry, I'm sorry, you need to reflect inward.

Do you want to hear a story about a room deciding that they finally believed "facts"?

It's 2006 at Nordstrom. When we executed a mail/holdout test, we could measure sales on our proprietary credit card within Nordstrom ... and at other brands. We could see if customers used the Nordstrom card to spend more at Neiman Marcus, or Saks, or Amazon. Was our catalog taking business from Neiman Marcus or Saks? No. It didn't impact those brands, brands we believed we competed with.

Amazon? In 2006?

• $10.00 spent at Amazon in the Mailed Group.
• $7.00 spent at Amazon in the Control Group.
• $3.00 per customer incrementally driven to Amazon.
• 15,000 credit customers per mail and holdout group.
• Results were easily statistically significant.

Want to know something even worse? Within the mail and holdout groups.

• $25.00 spent at Nordstrom in the Mailed Group.
• $23.00 spent at Nordstrom in the Control Group.
• $2.00 per customer incrementally driven to Nordstrom.
• 15,000 credit customers per mail and holdout group.
• Results were easily statistically significant.

Do you see what happened there?

Our brand-centric catalog at Nordstrom, in 2006 (we ended the ecommerce catalog program in mid-2005), DROVE MORE BUSINESS TO AMAZON THAN TO NORDSTROM. Imagine if that was happening to most catalog-centric brands (hint - it probably was)?

In the my early consulting days (2007 - 2012), I'd share this fact with catalog clients. The Executives would just sit there, dumbfounded, thinking about the consequences of driving more business to Amazon than to their own brand. Then they'd move on, pretending they never heard what I shared, because having to face the data was just a bit too much for the recipient of the message to accept.

However, when I shared this fact at Nordstrom in 2006, the data was immediately embraced. Nobody questioned it (though they frequently questioned mail/holdout results). They immediately understood the strategic implications illustrated by the data.

Nordstrom is still here.

Most catalog brands went out of business or were gobbled up by parent companies, thereby losing their identity.

I understand why you don't want to believe mail/holdout results. It's an outcome that requires sober reflection.

I understand why your paper rep doesn't believe the results. They "can't" be allowed to believe them.

I understand why your printer doesn't believe the results. They "can't" be allowed to believe them.

I understand why your boutique agency doesn't believe the results. They "can't" be allowed to believe them.

I understand why so many consultants don't believe the results.

But it doesn't mean the results are wrong.

There's always a group of individuals who chose to not believe data. Modern politics comes to mind. You don't convince people with "more data". You work around them. Because you are reading this, you're willing to accept what your customers are telling you - that's a good thing!