About sixteen years ago we started a program where if a customer spent $750 or more in a year the customer got to pre-select Anniversary Sale merchandise in our Nordstrom stores a week before the sale actually began.
We weren't allowed to create a holdout group.
We weren't allowed to perform a typical A/B test or any other form of experimental design.
We were expected to measure the incremental impact of the decision.
My team performed a reasonably simple analysis. They measured how much the $750+ group spent the year prior, then during Anniversary Sale the year prior. They compared this to how much the $650 - $749 group spent the year prior, then during Anniversary Sale the year prior. The change in performance was "attributed" to the new Anniversary Sale program.
Here's what the analysis looked like:
- 2005 Pre-Spend = $1,500.
- 2005 Anniversary Spend = $300.
- 2006 Pre-Spend = $1,550.
- 2006 Anniversary Spend = $380.
- 2005 Pre-Spend = $700.
- 2005 Anniversary Spend = $150.
- 2006 Pre-Spend = $725.
- 2006 Anniversary Spend = $155.
- (380/1550) / (300/1500) = 1.226.
- (155/725) / (150/700) = 0.998.
- (1.226 / 0.998) = 1.228.