FAQ From Catalog Mail / Holdout Posts

There were three posts (click here and here and here) ... and you had questions about the posts.

Question:  Attribution routines have to be better than catalog mail/holdout tests, right? They take all advertising channels into account, and they use machine learning, or so I've heard. So they're just better, and I don't have to execute catalog mail/holdout tests. Ok?

• If your vendor tells you that you do not need catalog mail/holdout tests in your attribution routine, please go find another vendor. The best attribution routines measure what happens when you do / do-not mail catalogs, and adjust accordingly.

Question:  We only have 90,000 twelve-month buyers, so we cannot execute holdout tests. Sorry, that's just life.

• You can execute holdout tests, you are choosing not to and are using a small twelve-month buyer file as the excuse. There is nothing stopping you from holding out 3,000 twelve-month buyers for six months. Nothing. Hold them out. Learn. Then run your business more profitably.

Question: We aren't going to run tests in every segment in every mailing, that's just stupid. What is the "right" test design?

• There is no right test design. There are your goals. And then there is a testing plan that allows you to see if your goals are being met or not. Most of my client base pick one of two paths ... they either test every single mailing (not as smart) or they test customer segments across 4-6 months (much smarter). But again, if you want to test every mailing, have at it, it's better than doing nothing (which is what many catalogers do).

Question:  What is the "right" sample size?

• There is no right answer to this question.
• Have too few customers in your test and your test results will bounce all over the place ... 38% organic, 83% organic ... and then you'll say "I don't trust testing" and you'll be right and it will be on you because you had too few customers in your test panels.
• If you only have 50,000 twelve-month buyers, you are better off going with 2,000 or 2,500 in your holdout group ... but then DO NOT measure the incremental value of each catalog ... only measure results 4-6 months later, in total.

Question: Our results are all over the board. Why?

• Remove outliers.
• Adjust all orders in the lowest 5% and highest 5% to the lowest 5% and highest 5% and your results are going to look much more consistent.

Question: After eight weeks, our test results change. Why is that?

• The role of prior catalogs diminishes after eight weeks. Focus on that third month and beyond. And this is why you focus on long-term results.

Question:  My vendor says you are nuts and that clients don't need to execute tests.

• It's easy for a vendor to point out what everybody else does right/wrong in a private conversation with you with the doors closed.
• Ask your vendor to point out where they publish how their methods generate more profit for clients than Mail / Holdout tests ... ask them to show you where you can find their published works, for free, shared for the world to see, ok? If the vendor doesn't even have the courage to publish their findings (for free) and earn attention that leads to new clients, how good can their findings be?

Question: E-commerce brands are using print all over the place, and they're not measuring via mail / holdout tests, so that means we're fine too, am I right?

• No.
• There are two reasons e-commerce brands are not using mail / holdout tests results. They either don't know they should, or the are being bamboozled by their vendor to demonstrate that they are generating $11.00 per customer per mailing (which will cause the e-commerce brand to mail more catalogs, which gets vendors paid even more). It is sad when e-commerce brands are being bamboozled by their vendor partners and don't know they're being bamboozled.

Question:  I executed six tests last year. Here are the organic percentages I measured ... 50% ... 58% ... 61% ... 39% ... 43% ... and 54%. What is the organic percentage I should use?

• Average your results ... 51%.
• When you execute a test, you are getting an outcome that fits within a normal distribution of potential outcomes. This is why you have to repeatedly test, so that you find out what a reasonable outcome actually is.

Question:  What does it mean when our organic percentage was 50% three years ago, then 54% two years ago, then 57% one year ago, and is now 60% today?

• It means that your business is becoming less and less dependent upon catalog marketing.
• It means you need to have an honest conversation with your Executive Team.
• Run scenarios and tell your Executive Team how much of a profit difference there is between what "as-is" strategy yields and what could be done using an ever-increasing organic percentage.

Question:  How do I apply the organic percentage to individual segments?

• Let's assume you already measure the organic percentage for online customers and traditional catalog shoppers and already know that there is a big difference.
• Ok, say a segment generated $3.00 during an eight-week period, and you know from prior testing that the segment is 60% organic. This means that 40% is catalog-caused. 
• Your calculation is $3.00 * 0.40 = $1.20 catalog driven demand.
• Your profit calc, assuming 40% flow-through rate and a $0.60 ad cost is $1.20 * 0.40 - $0.60 = ($0.12). You lost $0.12. The mailing (to this segment) was unprofitable.
• Your vendor will use the following equation ... $3.00 * 0.40 - $0.60 = $0.60 ... PROFITABLE! Your vendor will tell you to keep mailing.
• See how this stuff works? Your paper rep, your printer, your boutique agency, your merge/purge house, your database provider ... they all generate more money if you use the fraudulent $3.00 figure ... you generate more money if you use the accurate $1.20 figure derived from Mail / Holdout test results.
• This is the very reason why some in the vendor community steer you away from Mail / Holdout test results. You hurt their bottom-line when you measure things the right way.

Question: If you stop mailing for a year or more, the organic percentage gets worse. This is proof that catalogs work, right?

• This is proof that you shouldn't stop mailing altogether. Mail 1x or 2x per year instead of 24x per year and you'll see a different outcome.

If your vendor partners are quoting numbers to you that are not derived from Mail / Holdout results, please question your vendor partner why they are purposely misleading you? They might mislead you because they don't honestly know that they should be using Mail / Holdout results. That's fine. Then teach them why they are not providing accurate results. I'm 100% ok with that.

If your vendor partner is purposely misleading you (and it comes up all-the-time in my work), have a serious discussion about "why" your vendor partner is misleading you on purpose. Are they frightened that they will struggle to stay in business if they don't make catalog performance look great? That happens, and often. I understand that one as well, and am forgiving of that possibility.

But if your vendor partner is purposely misleading you so that they can be more profitable while you are less profitable, please, go find another vendor partner. It's hard enough just to stay in business these days, much less have a third-party skim your profit from you so that they can make more profit themselves.

And yes, readers, you already know who falls into that last paragraph ... we all know who falls into that last paragraph. If we keep working with those folks when we know they are ripping us off, then we are the problem, right?