Yeah, yesterday I got frustrated about sample sizes. I'm so tired of people who do not understand math using math to make a point that is vendor-friendly so that vendors make money. It's a form of lying, it is misleading, and it is corrupt.
So let me take you all the way back to Lands' End, in 1991. Yeah, 31 years ago. We were executing tests with 5,000 customers in a cell, and the test results were ALL OVER THE MAP. Not consistent.
I was asked to determine what an appropriate test sample size should be.
I measured the variance of test results at different demand-per-customer levels. There was a clear, linear relationship.
- -188 + 192*(demand-per-customer).
- q = SQRT(2*(-188 + 192*(demand per customer)/(sample size))).
- Sample Size = 5,000: Q = 0.68. 2*Q = $1.36.
- Sample Size = 10,000: Q = 0.48. 2*Q = $0.96.
- Sample Size = 25,000: Q = 0.30. 2*Q = $0.60.
- Sample Size = 50,000: Q = 0.22. 2*Q = $0.44.
- Sample Size = 75,000: Q = 0.18. 2*Q = $0.36.