Reader Question

This one came up numerous times last week.

Question:  "Why in the name of Don Libey are you talking about catalog mail/holdout tests? Seriously! It's 2025. I executed mail/holdout tests at Fingerhut in 1988. What is wrong with you?"

Allow me to tell you a story. I spoke at a conference a few years ago. I was asked to share what I've learned from executing and analyzing a thousand or more mail/holdout tests over a thirty-five year career (and not just from catalogs - email mail/holdout tests - search geography-based on/off tests - display on/off tests, you get the picture).

I mean, it might be thousands of tests I've been associated with, each including thousands or tens of thousands of customers per test ... tens of millions of data points. I doubt there is any person on planet Earth who executed and analyzed more tests applied to a print-centric business than I have. 

So I shared what I learned. I could tell that the audience didn't like what I revealed. Long faces. Heads shaking "no" left-to-right and right-to-left. People not taking notes.

What I was sharing, of course, was the fact that the attendees were on the precipice of "re-inventing" their businesses. The audience didn't want to learn that. The audience wanted to learn that they could keep doing what they loved doing.

I finished my presentation ... tepid applause. A paper professional immediately pulled me aside and told me he couldn't let my message ever get to his clients because my message would, and I quote, "take food off of his dinner table".

That's when the next speaker took the stage ... a person representing survey results from a print-centric agency. The speaker immediately attacked my presentation. He pointed to me in the distance, from the stage ... "He's quoting tests. Tests have all sorts of error and variability associated with them. I'm about to share facts with you. Yes, facts! We surveyed actual customers, and we know the truth." The audience bubbled with enthusiasm, as the speaker shared the results of a survey with a few hundred respondents ... a survey that was clearly designed to communicate the intangible value of print.

At the end of the presentation, the audience vigorously applauded. They were happy!

I realized I had been "set up". My presentation was there to communicate a point of view (the truth), but was then followed by an hour-long message that the conference organizer wanted the audience to hear. Within an hour, the audience perspective was reinforced. #printisback. All based on a survey of three hundred individuals. Positive vibes. Three hundred individuals were valued more than the millions of customers I'd analyzed via mail/holdout tests.

I analyzed a mail/holdout test for a client, a test showing that the catalog was wildly unprofitable. Their agency pushed back, suggesting that profitability wasn't the right metric, that I should be looking at the fact that engagement was significantly boosted (it was - website traffic increased nicely).

It's October 2025. Business owners and Executives cannot run businesses off of vibes and print engagement. Social engagement? Yeah, have at it, costs are minimal. Print is held to a different standard due to the prohibitive costs associated with print.

Execute mail/holdout tests. Learn. Adapt.