Way back in the early 1990s at Lands' End, we had seven different business units that marketed to customers, either through standalone catalogs, or though pages added to catalogs.
As growth became more and more difficult (pay close attention online marketers ... your world is heading in this direction), management elected to mail targeted catalogs to targeted customer segments.
In other words, a Mens Tailored catalog concept was developed, with a half-dozen or more incremental catalogs mailed to customers who preferred Mens Tailored merchandise. A Home catalog concept was developed, with nine or more incremental catalogs mailed to customers who preferred Home merchandise.
Seven concepts were developed. Each concept was growing.
But the core catalog, the monthly catalog mailed for three decades, was not really growing anymore. And total company profit (as a percentage of net sales) was generally decreasing over time.
Something was amiss.
We studied the housefile, and learned that the "best" customers were being "bombed" by catalogs ... upwards of forty a year. Every business unit, making independent mailing decisions, mailed essentially the same customers. And all of our metrics, when viewed at a corporate level, indicated that customers were not spending fundamentally more than they spent several years ago when the new business concepts didn't exist.
So we developed a test. We selected ten percent of our housefile, and created seven columns in a spreadsheet. We randomly populated each column with the words "YES" or "NO', at a 50% / 50% proportion. Each business unit was assigned to a column. When it came time to make mailing decisions for that business unit, we referred to the column assigned to the business unit. If the word "NO" appeared, we did not mail the customer (if the customer qualified for the mailing based on RFM or model score criteria).
In statistics, this is called a 2^7 Factorial Design.
There are two reasons for designing a test of this nature.
- Quantify the incremental value (sales and profit) that each business unit contributes to the total brand.
- Identify, across customers segments, the number of catalogs a customer should receive to optimize profitability.
- Each catalog mailed to a customer drove less and less incremental increases in sales. If a dozen catalogs caused a customer to spend $100, then two dozen catalogs caused customers to spend $141, and three dozen catalogs caused customers to spend $173. The relationship roughly approximated the Square Root Rule you've read so much about on this blog.
- Each business unit, on average, was contributing only 70% of the volume that company reporting suggested the business unit was contributing. In other words, if you didn't mail the catalogs, you'd lose 70% of the sales, with customers spending 30% elsewhere.
Take a look at the table below, one that illustrates the profit and loss statement reported by finance, and one that applies the results of the test.
| Test Results Analysis | Finance | From | |
| Reported | Test Results | ||
| Demand | $50,000,000 | $35,000,000 | |
| Net Sales | 82.0% | $41,000,000 | $28,700,000 |
| Gross Margin | 55.0% | $22,550,000 | $15,785,000 |
| Less Marketing Cost | $9,000,000 | $9,000,000 | |
| Less Pick/Pack/Ship | 11.0% | $4,510,000 | $3,157,000 |
| Variable Profit | $9,040,000 | $3,628,000 | |
| Less Fixed Costs | $6,000,000 | $6,000,000 | |
| Earnings Before Taxes | $3,040,000 | ($2,372,000) | |
| % Of Net Sales | 7.4% | -8.3% |
The test indicated that what appeared to be highly profitable business units were actually marginally profitable, or in some cases, unprofitable. In this example, the business unit is "70% incremental", meaning that if the business unit did not exist, 70% of the sales volume would disappear, while 30% would be spent anyway by the customer, spent on other merchandise.
Imagine if you were the EVP responsible for a business unit that appeared to generate 7.4% pre-tax profit, only to have some rube in the database marketing department tell you that your efforts are actually draining the company of profit?
Why Does This Matter?
This style of old-school testing (which is more than a hundred years old, with elements of the testing strategy now employed aggressively in online marketing) tells you how valuable your marketing and merchandising initiatives truly are.
Catalogers fail to do this style of testing, not realizing that a portion of catalog driven sales would still be generated online (or in other catalogs). In 2008, most catalog marketers are grossly over-mailing existing buyers. Catalog Choice, in part, exists due to catalogers mis-reading this phenomenon.
E-mail marketers seldom execute these tests, not realizing that in many cases almost all of the sales would still be generated online. E-mail marketers, ask your e-mail marketing vendor to partner with you on test designs like the ones mentioned in this article. You may be surprised by what you learn!
Online marketers are more likely than most marketers to execute A/B splits at minimum, with some executing factorial designs. Many online brands evolve in a Darwinian style, fueled by the results of factorial designs. Online marketers know that you make mistakes quickly, and you correct those mistakes quickly.
Web Analytics folks have the responsibility to tell management when sku proliferation no longer contributes to increased sales. It is important for Web Analytics folks to lead the online marketing community, shutting off portions of the website in various tests to understand the incremental value of each additional sku.
What are your thoughts on this style of testing? What have you learned by executing tests of this nature?
Great post Kevin, the #1 issue going forward for onliners.
ReplyDeleteA lot of web analysts feel since they don't have access to the data in "back end" systems, this kind of customer-level analysis is not their job. Indeed, many have job directives to maximize canpaign results, not customer profitability.
What we really need to get moving on the next level is:
1. Really curious web analysts who will cross over an start working with the back end data, perhaps even on the sly
2. Management folks who insist on understanding the full story
Let's keep fighting the good fight!