1993: Testing As A Strategic Weapon

Five popular songs from 1993:

• "I Will Always Love You" - Whitney Houston
• "Whoomp, There It Is" - Tag Team
• "I Can Help Falling In Love With You" - UB40
• "That's The Way Love Goes" - Janet Jackson
• "Freak Me" - Silk

I'd guess this happens at your business as well as at Lands' End, way back in 1993.  You have a merchandising team that must increase sales, or they get fired.  Yup, merchants have a fun job, with much more pressure than the average employee. 

So your merchandising team introduces a new product line.  And the numbers look great. Woo-hoo!

Except for one little problem.

The merchants responsible for the existing product lines are experiencing flat sales, or are experiencing sales declines.  Well, this can't be their fault, can it?

At Lands' End, there were two camps in 1993:

• Camp #1 = The core business was the engine behind the entire company, without it, everything fails.  New business units were stealing demand from the core business.
• Camp #2 = New business units were the growth engine of the company, without them, the business would experience flat or declining sales.

Here's a little hint - it is very tough to use existing data to prove that camp #1 or camp #2 is right. You need a test.

Why set up a test?  Well, we had all of these bizarre, circular arguments.  For instance, a customer might purchase from the core catalog in January, then purchase from a Home catalog in October.  The core business folks would argue that the October order cannibalized a core business order - the Home team (Home merchandise, pillows, comforters) would argue that without the October Home order, the customer would lapse and become unproductive.  The arguments would simply swirl around, without resolution - and no amount of data pro/con would change any viewpoint.  It was almost like a religion - folks had faith - and you couldn't use facts to change faith.  You needed a test.

What did we design, at Lands' End, to answer this question?

• Step 1 = Every customer number ending in "3" would be included in our test.
• Step 2 = We randomly created 1/0 flags ... for each business unit.  If the customer received a "1", the customer could be mailed catalogs if eligible.  If a customer received a "0", the customer would be suppressed from receiving catalogs from that business unit for the next twelve months.

Yes ... twelve months.

Almost nobody has the courage to execute a test of this magnitude.  Would you have the courage to not send an email to a customer for a year, to prove that email marketing truly worked?

Almost everybody should have the courage to execute a test of this magnitude.  How else are you going to accurately measure the impact of a business unit without withholding the business unit from customers?

Testing is a strategic weapon.  Stop testing nonsense (i.e. large orange button vs. small green button ... or 30% off plus free shipping vs. free shipping plus 30% off).  Test something that actually matters to your business!  In 1993 at Lands' End, we implemented a year-long test.

And I've got a feeling that, in 1994, what we learned would cause problems!