tag:blogger.com,1999:blog-32202893.post3035728219743221046..comments2019-06-01T11:18:34.003-07:00Comments on Kevin Hillstrom: MineThatData: A/B Testing Gone BadUnknownnoreply@blogger.comBlogger14125tag:blogger.com,1999:blog-32202893.post-1696132824279306922013-04-26T08:42:22.336-07:002013-04-26T08:42:22.336-07:00Once you roll up the data, you lose the ability to...Once you roll up the data, you lose the ability to measure variance, and therefore, make it impossible to calculate a statistical test.<br /><br />There are ways to cheat, to estimate variance, but that gets kind of advanced for most people.MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-31829868824755993792013-04-26T08:40:09.204-07:002013-04-26T08:40:09.204-07:00Thanks Kevin. I guess you need the raw data though...Thanks Kevin. I guess you need the raw data though right? I was wondering if there was a test that could measure significance based on rolled up # visitors participating in each cell and the revenue per cell.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-32202893.post-1695439452224846062013-04-25T11:28:26.756-07:002013-04-25T11:28:26.756-07:00I'm fine with a standard t-test ... I use SPSS...I'm fine with a standard t-test ... I use SPSS software to analyze test results. I can get revenue per visitor of $10 or more, doesn't matter. Online calculators are suspect.MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-31777483526950619272013-04-25T11:16:13.117-07:002013-04-25T11:16:13.117-07:00Hi Kevin, which statistical test do you use to eva...Hi Kevin, which statistical test do you use to evaluate the difference in RPV? Most of the ttests used by A/B significance calculators do not work when RPV exceeds $1. Is there a particular test that should be used?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-32202893.post-36613853645876510342012-08-14T15:53:03.804-07:002012-08-14T15:53:03.804-07:00I use SPSS to do almost all of my statistical work...I use SPSS to do almost all of my statistical work. SAS works well, too.MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-27751470172367136182012-08-14T15:43:07.779-07:002012-08-14T15:43:07.779-07:00Hi Kevin,
Can you explain what software or statis...Hi Kevin,<br /><br />Can you explain what software or statistical models you use to run the A/B conversion rate tests against the statistical significance of dollars per visitor?<br /><br />I recognize the need for that, but I am not aware of software that can do that.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-32202893.post-31644630128994189522010-07-24T10:32:20.376-07:002010-07-24T10:32:20.376-07:00Dan --- true enough. If you pull random samples f...Dan --- true enough. If you pull random samples from your original population and measure the variance, then do a simple histogram of the variance, you'll know!MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-53830328880077412682010-07-24T07:40:07.412-07:002010-07-24T07:40:07.412-07:00If you have the ability to calculate profit or mar...If you have the ability to calculate profit or margin you should do that as well, especially if you are testing discounts. Be careful with a t-test as well as it assumes that you will have a normal distribution around the value you are measuring which may not always be true.dannoreply@blogger.comtag:blogger.com,1999:blog-32202893.post-65487331771278564422010-07-22T06:28:41.767-07:002010-07-22T06:28:41.767-07:00Hi Justin, thanks for the comment. As a practitio...Hi Justin, thanks for the comment. As a practitioner, you get to see what happens simply by hand-calculating the metrics, not by trusting a software product.<br /><br />And, as a result, you get to learn more, too!MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-58482867325014833522010-07-21T23:45:10.807-07:002010-07-21T23:45:10.807-07:00Kevin, your analysis is spot on with most of the t...Kevin, your analysis is spot on with most of the test results I myself have witnessed.<br /><br />Google website optimizer's lack of tracking the $ per conversion is only further evidence that adding this variable makes running a/b tests significantly more variable, therefore making their "product" harder to use. However this is leading to tons of false conclusions for retailers. <br /><br />Nearly everytime GWO tells me I have a significant winner, when I factor in $ per conversion the winner is far from significant.Justin Palmerhttp://www.palmerwebmarketing.com/blog/noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-63778501044533866792010-07-21T10:09:26.308-07:002010-07-21T10:09:26.308-07:00Provide the audience with a dataset of 20,000 cust...Provide the audience with a dataset of 20,000 customers involved in a test, 10,000 seeing one landing page, 10,000 seeing the other landing page. <br /><br />Show the statistical test for conversion rate, and for $ per visitor. Show the audience the variance and standard errors calculated for each metric.<br /><br />You have the data to do this!!MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-84680536871430273452010-07-21T09:53:48.680-07:002010-07-21T09:53:48.680-07:00Your point that tracking revenue is important is a...Your point that tracking revenue is important is a good one, Kevin. It's especially important for catalog eCommerce sites (and less so for single product eCommerce landing pages where, in many cases, average order value may have low variability).<br /><br />However, this statement doesn't seem accurate: "Every time you run one of those software-based automated conversion rate A/B tests that measures responses until one outcome is statistically significant, you guarantee that measurement based on dollars per visitor is not going to be significant". <br /><br />Conversion rate and dollars per visitor are not inversely correlated. When we track tests using conversions and revenue per visitor, we often see the same variation producing the best results for *both* metrics. They can both be calculated for the same test.<br /><br />Your conclusion is also a little over dramatic when you say, "this partially explains why a decade of conversion rate optimization has yielded websites that, by and large, convert fewer customers than in the past." More likely, it's because companies are testing inconsequential tips & tricks (think button color), rather than analyzing the persuasive experience of the visitor.<br /><br />We also find that any testing is better than no testing. Once a company gets addicted to making data-based design and content decisions they are taking the first step on the path to ever-increasing sophistication in their metrics. <br /><br />I would encourage a company to get started with testing and improve from there. Avoid the "paralysis by analysis" and insecurity from thinking that you have to boil the ocean to get any value from testing.Chris Gowardhttp://www.widerfunnel.comnoreply@blogger.comtag:blogger.com,1999:blog-32202893.post-40260067274976898592010-07-21T07:26:59.931-07:002010-07-21T07:26:59.931-07:00It is there now ... depending upon the browser you...It is there now ... depending upon the browser you are using, it either appeared or did not appear, sorry.MineThatDatahttps://www.blogger.com/profile/14014200122021988374noreply@blogger.comtag:blogger.com,1999:blog-32202893.post-34199435851570477442010-07-21T07:14:24.878-07:002010-07-21T07:14:24.878-07:00I think you're missing a column of data in you...I think you're missing a column of data in your table (control group spend)?Anonymousnoreply@blogger.com