As an example, assume a multichannel organization mails a catalog to a housefile list of 1,000,000 names. The database marketer chooses the best 1,100,000 households, and randomly splits them into two groups. The "A" portion of the test are the 1,000,000 households who receive the catalog. The "B" portion of the test are 100,000 households who will not receive the catalog.
Maybe a month after the in-home date, the database marketing analyst is prompted to analyze the results. Within each group, the 1,000,000 who received the catalog, and the 100,000 who didn't receive it, the analyst calculates the average net sales within the catalog/telephone channel, the online channel, and the retail channel.
Here are sample results:
| Quantity | Telephone | Online | Retail | Totals | |
| Received Catalog | 1,000,000 | $6.00 | $8.00 | $21.00 | $35.00 |
| Did Not Receive Catalog | 100,000 | $2.50 | $7.00 | $19.50 | $29.00 |
| Incremental Lift | $3.50 | $1.00 | $1.50 | $6.00 |
In this example, the catalog drove an incremental $3.50 per customer to the catalog/telephone channel, $1.00 per customer to the online channel, and $1.50 per customer to the retail channel, for a total of $6.00 incremental sales per customer.
Because we mailed 1,000,000 households, the total net sales attributed to this mailing is 1,000,000 * $6.00 = $6,000,000.
Some vendors advocate a different methodology --- they advocate allocating any online and retail order generated during the time the catalog was active to the mailing of the catalog. This results in a gross over-estimation of the importance of the catalog. Please don't go down this path.
A similar methodology can be used to test multiple marketing activities at the same time. Assume an e-mail campaign was mailed to the opt-in portion of this audience. Within this audience, you randomly assign customers to one of four test segments. Here are some sample results.
| Quantity | Telephone | Online | Retail | Totals | |
| Catalog + E-Mail | 400,000 | $5.50 | $8.50 | $21.25 | $35.25 |
| Catalog Only | 50,000 | $6.00 | $8.00 | $21.00 | $35.00 |
| E-Mail Only | 50,000 | $3.00 | $8.10 | $18.65 | $29.75 |
| No Catalog, No E-Mail | 50,000 | $3.50 | $7.00 | $18.50 | $29.00 |
Tests like these yield interesting and intriguing results. Notice that the best strategy for the catalog/telephone channel was to mail only a catalog. The best strategy for the online channel was to mail a catalog and an e-mail. The best strategy for the retail channel was to mail both a catalog and an e-mail.
Statisticians can assist with significance tests, if you feel that is appropriate. It is more important to simply execute tests of this nature, and learn how all of your marketing activities interact with each other. What you learn about how marketing activities and channels interact with each other within our multichannel ecosystem may surprise you.
Hi Kevin,
ReplyDeleteLike the blog! On this test design subject, great way to measure lift generated by additional mailings, especially liked to see lift by channel.
Would be interested in your thoughts on the inverse of this testing:
If the % Lift is 20% or +$6 per customer by mailing the catalog;
by NOT mailing the catalog, will I only be putting 20% at risk. By NOT spending the Ad $ can I still generate 80% of the sales?
Ad expense is an important variable that is not addressed in these tests.
Regards,
Tom
Hello Tom, thanks for the good feedback!
ReplyDeleteMarketers should run a profit and loss statement on the incremental $6.00 of sales across channels.
If the $6.00 of volume delivers enough profit, or, if the loss is less than the long-term profit generated by these customers, then the catalog is worth mailing.