November 17, 2006

Williams Sonoma: Incremental Online Sales and Matchback Analysis

Williams Sonoma always does a nice job of sharing fun facts with the public. In their third quarter earnings release, they state that "55% of online revenues are generated by customers who recently received a catalog."

This is always an interesting topic of debate in the database marketing world. Williams Sonoma does not specifically state which of two popular analytical methods they use to measure this metric.

Most popular, and most vigorously argued against by the analytically adept, is the method of attributing every online order to the catalog channel, if a customer recently received a catalog. The theory behind this technique (often called a "matchback analysis") is that the catalog inspired the order. Many vendors promote this methodology, and for good reason. The technique can overstate orders attributed to mailed catalogs, and vendors have a vested interest in promoting paper as a viable means of profitable marketing. Critics will argue that if you mail your entire housefile, this methodology will cause you to attribute every single online order to the mailing of the catalog. Critics will also argue that if you mail every housefile name a catalog, and send every housefile name an e-mail, the methodology completely breaks-down, rendering the analysis useless.

Less popular is the method of an "A/B" split. The marketer randomly splits her mail list into two halves. 50,000 customers receive the catalog, a like group of 50,000 customers do not receive the catalog. Several weeks after the in-home date, the marketer measures total sales in the mail group and control group, in both catalog and online (and, where applicable, retail) channels. This method tends to provide much less-optimistic answers than the "matchback analysis". Critics will argue that this methodology cannot produce reliable results due to sampling error issues.

Which methodology do you believe is more appropriate for allocating online orders to the marketing channel that drove the order?