The Report Lies To You

Non-catalog readers, take the day off.

Once again, I share these images to show you just how wonked-up AI actually is. Spelling errors. Missing legs. If it's this wonked-up with a simple cartoon, imagine what else the gurus are trusting that in reality is completely borked?

But I digress.

Last week a member of the #printisback community on LinkedIn decided that #brands need to go back to 1996, as if J. Peterman were about to publish the Urban Sombrero on the cover.

The root of the lie being told to you is called "Matchback Reporting". Matchbacks ignore incrementality. Those who believe in Matchback Reporting ignore reality.

Time for a parable. Remember when I told you a member of the retargeting community took me to lunch, and for the cost of a Caesar Salad wanted me to convince the Management Team at a large retail brand that because 95% of ecommerce customers saw at least one of his ads he deserved credit for 95% of ecommerce orders (over $300,000,000 in annual sales) and he wanted a percentage of each transaction. If he presented those facts to you, a smart cataloger, you'd say "hey, Goober, get out you Lemonhead." Then you'd go back to your cubicle, pull out your Matchback Report, and perform the EXACT SAME ANALYSIS AS GOOBER DID and treat it as Gospel. You'd think that Goober was an idiot, you'd view you as a savvy marketer. And yet? You're the same person, doing the same thing, sans Caesar Salad.

Let's assume that your catalog is active across a four-week period. Let's assume you were going to mail your entire twelve-month customer file. You randomly select 50,000 customers to be in the mailed segment, you randomly select 50,000 customers to be in your no-mail / holdout segment.

You then sum all demand across all marketing channels for the four weeks when the catalog is active. Compute an average (i.e. divide each sum by the 50,000 customers in the segment). You'll produce a table that looks like this:

This is the way test results generally look. If you don't mail a catalog, your call center is quiet as 78 year olds cannot call you if they don't have a catalog in their hand. Website / Direct Load is where most demand happens, most of the demand will happen if you don't mail the catalog. Email Marketing is usually not impacted by catalogs, though your mileage will vary. Search is a channel that is clearly impacted by catalogs - catalogs cause customers to search for competing products (both a strength and a weakness of catalog marketing because your dumb catalog drives your smart customers to the competition). Social is almost never impacted by catalogs - completely different audiences.

The magic in the table happens in the Total Demand column.

Your matchback reporting takes full credit for the $5.65 of total demand. It ignores what would have happened if the catalog were never mailed. If the catalog were never mailed, the $5.65 total becomes $4.25 ... not $0, but $4.25.

Here's what the p&l might look like for the $5.65 figure.

Everything looks good here ... you appear to generate $1.54 profit per catalog/book ... in modern parlance you generate a ROAS of 7.53. It's the 7.53 figure that the #printisback community on LinkedIn like to refer to ... it's a much higher number than that 3.88 figure you get for paid search or 2.97 for paid social or ... wait ... they never quote email marketing because email marketing has the best ROAS, period.

Remember - your holdout group did $4.25, not zero. Take the control / holdout group average ($4.25), divide it by the mailed group average ($5.65), and you get 75%.

This means that 75% of what is outlined on the matchback report is a lie. A fabrication. It would have happened had the catalog not been mailed ... and you know this is true because in the table YOU DIDN'T MAIL THE CUSTOMERS IN THE HOLDOUT GROUP!!!!

We cannot run the p&L on the $5.65 that is likely reported in your matchback report. We have to run it on the incremental total ... $1.40 ... which is (1 - 75% = 25%) of the $5.65 total.

The p&l changes, friends.

This is where things get really dicey. The incremental outcome (a loss of $0.18 per catalog) is REALITY. The matchback-reported totals column of $1.54 profit is FANTASY.

On LinkedIn, the #printisback community communicates FANTASY results to you, misrepresenting the outcome as reality.

It's pretty obvious why they'd do this.

• $5.65 per book / $1.54 profit = Best ROI, which means you should hire them.
• $1.40 per book / ($0.18 profit) = It might be time to shut down your catalog division.

When I communicate (via analysis of thousands ... seriously ... of catalog mail/holdout tests across 36 years) what a smart catalog brand should be doing, I'm generally derided by the #printisback community, and for good reason. I want you to do what is most profitable for your business, they want you to do what is most profitable for their business. Do you see the distinction there?

So, yes, the #printisback community is probably right ... I'm not one of them as they tell my clients.

But I'm on your side. I want you to be as profitable as possible. Matchback reports lie to you, and somebody has to communicate that to you.