tag:blogger.com,1999:blog-32202893.post4996758893581880280..comments2023-10-18T08:32:17.510-07:00Comments on Kevin Hillstrom: MineThatData: The Nordstrom Catalog Experiment of 2004 - 2006: An FAQUnknownnoreply@blogger.comBlogger5125tag:blogger.com,1999:blog-32202893.post-11518408551307163542011-07-12T05:57:44.244-07:002011-07-12T05:57:44.244-07:00Rishi --- let's say you have 250,000 twelve-mo...Rishi --- let's say you have 250,000 twelve-month buyers on your customer file. You take 25,000 customers, randomly selected, and split them in half. 12,500 receive all catalogs they normally would receive for the next three months, while 12,500 receive no catalogs. At the end of three months, you measure the incremental difference in sales between the group that was allowed to receive catalogs, and the group not allowed to receive catalogs. The incremental difference between the two groups is what catalogs actually delivered, in terms of ROI.<br /><br />Tanya --- most of our tests were only one month or three month tests, with a smaller number of tests run for six months.MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-83765735794474736232011-07-12T05:08:08.237-07:002011-07-12T05:08:08.237-07:00Back to question #1, how long did each hold-out te...Back to question #1, how long did each hold-out test last? Six months, 12 months?Tanyanoreply@blogger.comtag:blogger.com,1999:blog-32202893.post-73922793044989416042011-07-11T23:17:10.514-07:002011-07-11T23:17:10.514-07:00Can you explain what you do exactly in a holdout t...Can you explain what you do exactly in a holdout test?Rishihttps://www.blogger.com/profile/12411867035238965406noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-17878912035591288772011-07-11T22:06:55.690-07:002011-07-11T22:06:55.690-07:001 - We ran dozens of holdout tests, we knew very w...1 - We ran dozens of holdout tests, we knew very well what would happen. Holdout tests do not lie!<br /><br />2 - It is my opinion that you'll get reliable and often comparable results in B2B as in the Nordstrom example I cited. Not many people agree with me.<br /><br />3 - I don't think the results would have been that much different. We had an integrated database in 2002, and the results were similar in 2002, 2003, and 2004.MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-72632452011254366562011-07-11T21:37:11.687-07:002011-07-11T21:37:11.687-07:00That's an amazing story! I'm curious to k...That's an amazing story! I'm curious to know:<br />1. How long did you run the hold-out test before you were confident with the results and considered it complete?<br />2. Do you think that B2B catalog hold-out tests would have a similar outcome or is that type of customer not as "evolved" or accustomed to shopping online?<br />3. If you had run the Nordstrom hold-out test in 2000, do you think the catalog decision would have been the same? If no, why?Tanyanoreply@blogger.com