Mail / Holdout Tests

There isn't one topic that has earned me more universal hate in my career than the topic of mail / holdout test results.

I've deleted more words in this post than nearly any other post written in the past five years. Version after version written, saved, re-read, then deleted.

Pre-COVID, a person reached out to me to mention that they executed holdout tests for a full year. The holdout tests showed that the catalog had minimal impact on top-line sales. They discontinued catalog mailings. Top-line sales didn't change much (if I remember, there was a 7% sales hit). Of course, with fifteen percent of the expense structure eliminated, profitability was at an all-time high.

I doubt (prove me wrong) that there is a professional who has executed / analyzed more catalog or email mail/holdout tests than me. I've been doing it for nearly four decades.

Regardless, I receive a lot of pushback. Pushback falls into the following categories.

1. I don't believe the results of tests.
2. We can't lose sales because somebody wants to test something.
3. Our culture won't do anything with the results of a test, so why test?
4. I don't want to change.

I used to have contempt for #4 ... that has "changed" over time. Seven years ago a 58 year old told me that he was "just trying to get to the finish line" ... meaning he didn't want to overhaul how an entire company did everything as the final act in his career. I get that. Still means you need to do the work, but I understand why some Baby Boomers vetoed knowledge.

Tomorrow we'll talk about the impact of a mail/holdout test.