Apparently Amazon is "Doing It Wrong"

Over on LinkedIn, catalog agency folks and paper gurus are frustrated with Amazon ... they are criticizing Amazon for failing to "target" the recipients of the catalog properly ... openly soliciting Amazon for business in the process. Nobody ever executes catalogs "the right way" for these people.

I asked CoPilot to create another cartoon. Yes, AI has a spelling error ... extra credit for those who catch it ... but I want you to read what CoPilot added for text after drawing the image.

Email Test

Let's give you the same offer I made for catalog mail/holdout tests.

• For $900 per test, I'll analyze your email mail/holdout tests, showing you how customers interact with other channels when email doesn't exist.

Let's prove that your email marketing program has value (hint - it does) ... heck, you might learn that the majority of your e-commerce division's profitability is entirely due to email marketing (that's what many of my clients learn).

Contact me now (kevinh@minethatdata.com).

What Does An Email Mail/Holdout Test Look Like?

More than twenty years ago at Nordstrom we personalized every email we sent, and we executed mail/holdout groups. We knew that email was more productive than what our open/click/conversion metrics showed - we knew it was worth the marketing investment to personalize all of it and then generate another 20% on top of something that was already wildly profitable and not getting full credit.

We did that back in 2002. That's twenty-three years ago. Think about that for a moment.

You're emailing customers 10x a week, so you could hold out a group of customers for three to fourteen days, learning how customers interact with all of your channels.

After a few weeks, you average demand/sales by attributed marketing channel.

This is a very common outcome in an weekly email mail/holdout test. The two numbers to focus on are bolded in blue above.

• The group that received emails all week spent $0.50 per customer. This is what you'd normally report across maybe 5-10 campaigns, in total.
• The group that did not receive emails all week spent $2.00 - $1.54 = $0.46 per customer of incremental volume.

In other words, 92% of what this company commonly reports as email marketing revenue is actually email marketing revenue ... the other 8% would happen anyway if email marketing campaigns were stopped.

It's common to work with clients where email marketing might account for 15% to 20% of annual volume. That's a lot of volume! In this case, we'd multiply 15% to 20% of annual volume by 92% based on the test result, which means that email marketing is truly responsible generating A FORTUNE!

I've been in meetings where we are able to show that a company with seven percent pre-tax profit would be unprofitable without email marketing. You should see the eyes on the Finance folks when that fact is shared. And yes, somebody then says they can't believe the science behind a mail/holdout tests though they sure seem to appreciate the math behind clinical trials.

Email marketers almost never get the credit they deserve. A small number of under-resourced individuals do all this work and then present the findings via opens / clicks / conversions, which are metrics that your average employee struggles to be passionate about. Profit? At worst case, your Finance team becomes interested.

I've been in the meetings where the email service provider tells my client not to execute tests ... "oh you don't want to do that!". Why not? Seriously. Why does a vendor get to tell you what you should and should not learn (I know, you're going to say why does a rogue consultant get to tell you what you should and should not learn).

Did you know that when I was VP of Database Marketing at Nordstrom, back in 2003, my testing budget for marketing channels was 3% of annual e-commerce net sales? In other words, if we were a $400,000,000 e-commerce channel, we were willing to forego $12,000,000 in annual sales to learn how our customers interacted with retail stores, our website, and our catalog. Hint - that's how we learned we didn't need to mail catalogs anymore.

You want to prove your value to your company and then secure resources to do the stuff you've always wanted to do that nobody will give you money to execute because the people who have the money don't align with opens / clicks / conversions.

I worked with a client a decade ago who had an email professional who personalized all of her campaigns and generated what appeared (via tests) to be a 50% increase in the productivity of her campaigns. She didn't take the logical step to convert her numbers to profit ... so when she left the room one of the Executives looked at the Executive next to me and said "WHAT A NERD".

That's what you are up against. That Executive likely earned $250,000 a year plus an 80% bonus that allowed him to purchase a new BMW every year if we wanted to do so ... all because this young woman was generating most of the profit at her company. And he had to belittle her ... she's the problem.

No. She's the reason the brand was profitable.

Execute an email marketing holdout test.

Let's Try Something!

For the low cost of just $900, I will analyze your email mail/holdout test or your catalog mail/holdout test. I will show you how your catalog housefile matchback rules should change. I will tell you what your organic percentage is. I will tell you how profitable your catalog effort was. If it is an email mail/holdout test, I'll tell you if your email marketing program is being over-valued or under-valued.

For $900. Per individual mail/holdout test.

You have one (1) day to take me up on this offer. kevinh@minethatdata.com.

Go! Learn how your customers interact with your business. This stuff is fun!

Reader Question

This one came up numerous times last week.

Question:  "Why in the name of Don Libey are you talking about catalog mail/holdout tests? Seriously! It's 2025. I executed mail/holdout tests at Fingerhut in 1988. What is wrong with you?"

Allow me to tell you a story. I spoke at a conference a few years ago. I was asked to share what I've learned from executing and analyzing a thousand or more mail/holdout tests over a thirty-five year career (and not just from catalogs - email mail/holdout tests - search geography-based on/off tests - display on/off tests, you get the picture).

I mean, it might be thousands of tests I've been associated with, each including thousands or tens of thousands of customers per test ... tens of millions of data points. I doubt there is any person on planet Earth who executed and analyzed more tests applied to a print-centric business than I have. 

So I shared what I learned. I could tell that the audience didn't like what I revealed. Long faces. Heads shaking "no" left-to-right and right-to-left. People not taking notes.

What I was sharing, of course, was the fact that the attendees were on the precipice of "re-inventing" their businesses. The audience didn't want to learn that. The audience wanted to learn that they could keep doing what they loved doing.

I finished my presentation ... tepid applause. A paper professional immediately pulled me aside and told me he couldn't let my message ever get to his clients because my message would, and I quote, "take food off of his dinner table".

That's when the next speaker took the stage ... a person representing survey results from a print-centric agency. The speaker immediately attacked my presentation. He pointed to me in the distance, from the stage ... "He's quoting tests. Tests have all sorts of error and variability associated with them. I'm about to share facts with you. Yes, facts! We surveyed actual customers, and we know the truth." The audience bubbled with enthusiasm, as the speaker shared the results of a survey with a few hundred respondents ... a survey that was clearly designed to communicate the intangible value of print.

At the end of the presentation, the audience vigorously applauded. They were happy!

I realized I had been "set up". My presentation was there to communicate a point of view (the truth), but was then followed by an hour-long message that the conference organizer wanted the audience to hear. Within an hour, the audience perspective was reinforced. #printisback. All based on a survey of three hundred individuals. Positive vibes. Three hundred individuals were valued more than the millions of customers I'd analyzed via mail/holdout tests.

I analyzed a mail/holdout test for a client, a test showing that the catalog was wildly unprofitable. Their agency pushed back, suggesting that profitability wasn't the right metric, that I should be looking at the fact that engagement was significantly boosted (it was - website traffic increased nicely).

It's October 2025. Business owners and Executives cannot run businesses off of vibes and print engagement. Social engagement? Yeah, have at it, costs are minimal. Print is held to a different standard due to the prohibitive costs associated with print.

Execute mail/holdout tests. Learn. Adapt.

How Do Digital Channels Interact With Catalog Marketing Efforts?

Here's the test result we've discussed this week.

There is so much to digest within the table. The impact of catalog marketing on digital channels can easily be observed in the table. Your attribution rules clearly come into focus here.

Email Marketing:

• The catalog essentially has no impact on email marketing. About 4% of email demand is "caused" by the catalog.

Search Marketing

• A different situation. Half of Search Marketing demand is caused by the catalog ... the other half of Search Marketing demand happens independent of the catalog. Since the catalog causes half of Search demand, the catalog should get credit for the demand it pushed to Search ... and ... and, the catalog should be penalized for the ad cost in Search that wouldn't have to be spent had the catalog not been mailed.

Paid Social

• A similar story here. 60% of Paid Social demand still happens if catalogs are discontinued. The catalog should get credit for 40% of Paid Social demand and should have to absorb 40% of Paid Social expense.

The results of the test inform your attribution rules. Not just inform them, they determine your attribution rules!

So, that concludes our study of this one mail/holdout test. If you believe in doing the right thing for your company, you'll keep pushing this rock uphill. It will be an exhausting experience. It will be a worthwhile experience - protecting your company well into the future.

Question? (kevinh@minethatdata.com)

What Does It Mean?

Here's our test results.

We know that the catalog isn't generating profit as a whole. Now, if you run a dozen tests like this and get comparable results, you have a series of decisions to make, don't you?

At a segment level, your matchback results consistently lie to you.

At a segment level, after taking credit for just 30% of what the catalog appears to generate, you can't believe your own eyes, can you?

Each row represents a customer segment - the segment via matchback reporting (the Matchback column) converts demand to profit (the MB or Matchback Profit column). Only the segment generating $1.00 per book is unprofitable. That's the world many readers live in today.

However ... look at the "Test True Gain" column. Here we only take credit for the 30% of demand that the test tells us we can truly take credit for. Fewer than half of the segments are profitable.

Instead of being able to mail nine out of ten segments ... we can only mail four out of ten segments. Circulation would have to be cut by 1 - 4/9 = 55%.

This is the point in the discussion where catalog professionals become angry with me.

• "We can't cut circulation by 55%, why are we even mailing catalogs at that point?" Good question!
• "If we cut circulation by 55%, we'll lose sales, and we can't contract the top-line. No smart company achieves success by getting smaller".
• "What if you are wrong, Kevin?" Nobody ever wants to answer the question "What if the catalog professional is wrong?"

From here, there's all sorts of obscurification. 

• "We have to retest". 
• "I just don't believe you".
• "I'm not laying off half of my call center just to make you happy."
• "My catalog marketing agency told me to not listen to you, that test results are not a best practice given that matchback analytics have been around for twenty-five years."
• "My paper rep said he'd give me a discount to entice me to keep mailing catalogs. And the USPS is running a 10% off promotion. I'd rather band together with other catalogers and do what is right."

All of that, of course, ignores the reality of the situation.

• The catalog is not generating profit.
• The catalog professional could make decisions to make his/her company a fortune.
• What would the CFO think if she knew that the catalog professional was purposely not maximizing profitability?

Tomorrow, we'll wrap up this mini-series with a discussion of digital marketing channels.