Those of us who are in the business of allocating customer responses to advertising vehicles frequently use a great tool --- called the "matchback".
In a Matchback Analysis, the business looks at a customer who buys something, and tries to allocate that purchase to one of the most recent advertising strategies employed by the business. Some companies come up with complex business rules that are used for allocating orders to advertising vehicles.
A Matchback Analysis is SOOOO much better than doing nothing!
But the methodology is flawed. Especially for catalogers.
An example. On Monday night, I dropped my dog off for a week-long stay with his Mom (these two dogs know each other and acknowledge each other upon visits, which is neat to see). I needed to grab a bite to eat, and had only one fast choice nearby, McDonalds.
What inspired this Filet 'O Fish purchase was convenience and speed. Advertising did not play a role in this purchase.
Yet, in a Matchback Analysis, the analyst would attribute my purchase to any one of the ten McDonalds commercials I saw on Sunday.
Matchback Analysis will always cause folks to overestimate the impact of advertising, especially in the catalog world.
It is still good to use Matchback Analysis in your catalog-based marketing efforts, to gain an understanding of incremental online sales. A far more valid measurement strategy is to not mail catalogs to a small group of customers for six months or longer. Compare that group to an equal group of customers who receive catalogs. The results of this type of testing will significantly differ from Matchback Analysis ... and are much more accurate.
Your test for a matchback analysis in the catalog space reminds me of a humbling experience I had early in my career. We set aside a statistically valid control group and mailed our client's leisure travel promotion to a test segment of the house list. Three months later, the control group actually outpulled the test group by a slim margin (not one that was greater by statistically reliable degree, but STILL -- try making an ROI case for mailing the next catalog!).
ReplyDeleteThe lessons learned: 1.) Off travel seasons (such as the one we were in) amplify the myriad influencers on decisions to take vacations, and the decision about whom you will take them through, and 2.) Sometimes it doesn't pay to come into work in the morning.