January 13, 2009

When Metrics Lie

Our metrics continually lie to us.

Some are so easy to debunk ... "Multichannel Customers Are The Best Customers".

Some are much more complex.

Back in 1993, Lands' End was caught up in the "targeting" movement. In other words, the core monthly catalog was not driving large enough sales increases to please management and shareholders. As a result, management chose to create specialty catalogs, targeted to customers who previously purchased niches of product. What was once a catalog strategy that yielded thirteen catalogs, one every four weeks, became a catalog strategy that offered some customers thirteen monthly catalogs, nine Kids catalogs, nine Home catalogs, six Mens Tailored Merchandise catalogs, four Womens Business Merchandise catalogs ... you get the picture.

The finance department ran profit and loss statements for each business. As the businesses matured, the profit and loss statements indicated that these new, targeted catalogs were working really well, while the monthly catalogs were struggling. Here is an example (numbers dummied-up to protect the innocent):

Financial Profit And Loss Statement

Core Startup Line Total

Business Of Business Business
Demand $80,000,000 $20,000,000 $100,000,000
Net Sales $64,000,000 $16,000,000 $80,000,000
Gross Margin $32,000,000 $8,000,000 $40,000,000
Less Mkt Expense $16,000,000 $2,720,000 $18,720,000
Less Pick/Pack/Ship $7,360,000 $1,840,000 $9,200,000
Variable Profit $8,640,000 $3,440,000 $12,080,000
Less Fixed Costs $6,400,000 $1,600,000 $8,000,000
EBIT $2,240,000 $1,840,000 $4,080,000
% of Net Sales 3.5% 11.5% 5.1%

On the surface, the specialty catalog, the one targeted to prior customers of various merchandise assortments, performed well.

This didn't sit well with some of us. So we created year-long tests, where some customers did not receive specialty catalogs.

At the end of the year, we learned that one of the specialty businesses, if terminated, would hand sixty percent of the business back to the core monthly catalogs. In other words, the specialty catalog was cannibalizing the core catalog ... making the core catalog look bad, from a performance standpoint.

We worked with the finance department to re-state the profit and loss statements. Look at the difference a 60% cannibalization rate creates.

Actual Customer Behavior Profit And Loss Statement

Core Startup Line Total

Business Of Business Business
Demand $92,000,000 $8,000,000 $100,000,000
Net Sales $73,600,000 $6,400,000 $80,000,000
Gross Margin $36,800,000 $3,200,000 $40,000,000
Less Mkt Expense $16,000,000 $2,720,000 $18,720,000
Less Pick/Pack/Ship $8,464,000 $736,000 $9,200,000
Variable Profit $12,336,000 ($256,000) $12,080,000
Less Fixed Costs $6,400,000 $1,600,000 $8,000,000
EBIT $5,936,000 ($1,856,000) $4,080,000
% of Net Sales 8.1% -29.0% 5.1%

This profit and loss statement is one that reflects actual customer behavior. The targeted specialty catalog is truly only adding $8,000,000 of incremental volume, and is actually losing $1.9 million a year.

Needless to say, this information was not well received.

But the information represents the truth.

We're so quick to believe the metrics we read every day that we seldom question them. "Oh, this landing page had a 6.3% conversion rate, that's good, that must have worked!!". It is possible that something else caused the landing page to work, that the landing page itself wasn't successful, but that something we aren't measuring caused the success?

We learn that our metrics are wrong when we operate in a "testing culture". A testing culture demands that we ask questions, that we develop hypotheses, and that we test our hypotheses. All of the magic happens when different strategies are tested.

The KPI/Dashboard maven is right to want to create nine important metrics that dictate success --- most employees struggle with multidimensional issues and complexity.

But if you are reading this, you are not like "most employees". You are different. You are curious. You do not accept the version of truth the trade journals, bloggers, vendors, or consultants paint for you.

Truth isn't illustrated by a metric. Truth is multidimensional, and is always available to those who seek it.


  1. Excellent post Kevin! I've seen the same process unfold numerous times in the print DM world. A (naive) suggestion which sounds so good to some people goes "The big book isn't producing enough of an increase in profit anymore, we need to 'niche out' the book!". If the niche book is composed of incremental product that only is applicable to a sub set of the target universe, okay, lets test it, but if you simply strip out product from the big book, the approach will usually fail. The niche book still holds much of the cost in paper and postage but the radically lower sku count will non sustainable, as seldom are their customer clusters who ONLY buy from a distinct line (at least in the industries I’ve experienced).

    The concept of incremental impact seems straight forward, but it is surprising to me how many people either don't get it or fight it vehemently.

  2. It's probably a harder topic to conceptualize than I give it credit for.


Note: Only a member of this blog may post a comment.

Treasure Hunt

A few weeks ago, a reader said this: "Instead of talking about the gloom and doom of the death spiral, why not be optimistic and ...