May 24, 2007

Matchback Analysis Challenge

Catalogers --- here's what I'd like for you to do on your next catalog.

Step 1: Randomly select 10,000 customers from your mailing.

Step 2: Divide these customers into two groups.

Step 3: In the first group, code these customers as "mailed". Send these customers a catalog.

Step 4: In the second group, code them in your promotional history as "not mailed". Do not send these customers a catalog.

Step 5: After these catalogs have been mailed and results have been measured, send each group of customers to your favorite matchback vendor, be it Experian or Abacus or anyone else.

Step 6: DO NOT TELL EXPERIAN OR ABACUS THE DIFFERENCE BETWEEN THE TWO GROUPS. Simply perform the matchback analysis on each group.

Step 7: Carefully analyze the results in each group. Remember, the second group DID NOT RECEIVE A CATALOG. Therefore, every order that Experian or Abacus attributes to a catalog in this group represents an "overstatement", a "matchback bias" in web orders that are attributed to catalog mailings. Remember, these customers DID NOT GET A CATALOG, you're just communicating to your matchback vendor that they got a catalog, so that you can measure the "matchback bias" that exists in all of these analyses.

After you've done this little experiment, share the results with my audience. I'll be happy to publish what you've found, regarding "matchback bias". I think you'll be surprised by what you learn from this experiment. I think you'll see that you are over-stating the importance of catalog mailings at driving web sales.