Analytics Sunday: Which System Do You Run?

Do you know which "system" you run?

In football, there are many offensive and defensive systems.  Defenses run a 4-3 or a 3-4 scheme.  Offenses run a west coast offense, spread offense, spread option, option, wishbone, veer, run & shoot, I-formation, wing-T, on and on and on.

The system you run dictates the talent you need.  The system you run dictates your worldview, it dictates how you approach problems.

The same issues apply to the analytics community.  You may not know it, but you "run a system".  And your system, whether you know it or not, defines your relationship between your co-workers, your Executive team, and your public persona.

The most important part of defining your system is understanding what your primary area of concern is.  You can only align yourself with one of the following two statements.

1. Your primary concern is to understand how marketing campaigns perform.
2. Your primary concern is to understand how customers behave.

You do not measure marketing campaigns to understand customer behavior ... that is fool's gold.  Either you are trying to optimize marketing campaign performance, or you are trying to optimize customer spend.

And mind you, every analyst worth her salt uses pieces of every system.  But her first instinct, when any question arises, is to fall back on her primary system.


If your primary concern is to understand how marketing campaigns perform, then you are likely to run one of the following systems.




Segmentation:  The primary goal of the Segmentation system is to categorize customers into homogeneous groups, for the purpose of optimizing marketing performance.  Segmentation was popularized by the RFM, or "Recency - Frequency - Monetary" system.  RFM is seldom used to understand customer behavior ... you don't ever know what is in the mind of a customer when the customer is a 4-6 month, 2x+, $100 AOV buyer.  You don't know what is in the mind of a customer who is an "Existing Visitor" in Coremetrics.  You simply categorize people in like groups for the purposes of understanding the effectiveness of your marketing activities.  Segmentation has many cousins.  CRM blends automation with segmentation.  PRIZM clusters classify customers into similar lifestyle profiles.  Web Analytics is a modern fusion of segmentation in an OLAP environment.  If the primary goal of your analytics department is to classify customers for the purposes of measuring marketing activities, then you run a Segmentation system.

Optimization:  The primary goal of Optimization system is to use testing to significantly improve marketing performance.  The Optimization system is fundamentally different than the Segmentation system in approach.  The Segmentation system presumes that various segments outperform other segments, allowing the marketer to target a customer with a message.  The Optimization system presumes that testing strategies yield the optimal strategy to pursue within a customer segment.  Instead of believing that marketing is right, the Optimization system believes that testing methodology reveals truth.  If, when asked a question, you immediately think about "testing" the question against another strategy, then you run an Optimization system.  Optimization systems are highly popular among the online marketing community, as well as many in the database marketing community.


Prediction:  The primary goal of the Prediction system is to create statistical models to explain customer response to marketing, or to target individual customers for marketing efforts.  This system is different, in that it believes in one-to-one communications automated by mathematical equations.  Database Marketing, by and large, is a fusion of Segmentation and Prediction, with a skew toward Prediction.  Online Personalization is an outgrowth of Prediction.  E-Mail "relevancy" requires a strong "Prediction" system.  If you build statistical models to rank-order customers for catalogs, you might run a Prediction system.  If you create statistical models to measure online influence, you run a Prediction system.  If you build statistical models to identify customers for telemarketing campaigns, you run a Prediction system.  If you build statistical models to identify the impact of a loyalty program, you might run a Prediction system.  The key is your "first instinct" when approached with a marketing problem ... if you immediately think about models, you run a Prediction system.




If the primary purpose of your marketing activities is to understand the impact of customer behavior on a business, independent of marketing activities, then you are a bit different.  You run a Forecasting system.


Forecasting:  The primary goal of the Forecasting system is to holistically understand how customers evolve and change over time.  Those who run a Forecasting system move outside of marketing campaigns, instead seeking to link macro-level customer habits to current and future events.  The style of database queries is different ... using pre/post periods to understand customer behavior, coupled with forecasts/projections for future customer activity given prior trends.


Now, obviously, you use all four systems in your analytics work.  You skew toward one system, using elements of all systems.

That being said, why does any of this matter?


Well, it turns out that the system you run needs to be in harmony with the system that your Executive team believes in.


For instance, how would you approach answering the following question:

• Conversion rates on your website are down eleven percent in 2011.  What is causing this problem, and what should be done to fix it?

Each analytics system approaches the question from a different perspective.

• Segmentation folks look at conversion rates among buyers, existing visitors, new visitors ... they analyze conversion performance by traffic sources, trying to identify the underlying problem.
• Optimization folks attempt to solve the problem first, looking to test different strategies that might resonate with the customer ... why look to the past when one can fix the problem by influencing the future?
• Prediction folks attempt to solve the problem first, by targeting those who are visiting the site with relevant content.  Or, this system might "equalize" customers using a statistical model, in order to understand underlying problems in past marketing campaigns.
• Forecasting folks combine macro trends with changes in customer behavior across channels, looking to identify trends (i.e. Mobile) that influence the future trajectory of website performance, forecasting the impact of lower conversion rates going forward.

Say your team runs the Prediction system.  Say your Executive team responds to the Segmentation system.  Well, you've got a disconnect, don't you?  Your Executive team wants to understand marketing performance, and you're out there promoting a personalization plan to boost conversion rates.  Ooops.  Or worse, say you promote a Forecasting system ... you are constantly talking about the impact that changes in long-term customer behavior caused by search/social/mobile will have on the business, while your Executive team craves campaign-based KPIs and Dashboard results of marketing campaigns.


Take a moment, and answer each of the following two questions:

1. When deciding who should receive a marketing promotion, do you like to segment customers into like groups (1), or do you like to create models that predict how likely customers are to respond to marketing messages (9)?
2. When considering a new marketing strategy, do you like to run a series of simulations that predict how customers might behave in the future (1), or do you like to run a series of tests that reveal an optimal marketing strategy to employ against your customer base (9)?

Plot the answer to the first question on the following chart ... (1) on the left, (5) in the center, (9) on the right ... then for the second question, plot a (1) on the bottom, (5) in the center, (9) on the top.


Repeat this exercise for the perceived answers your Executive team would provide you.

If you and your Executive team plot out in a similar manner, there's a good chance that you are aligned, that your Executive team understands what you are trying to accomplish.


If you and your Executive team are far apart on this graph, well, that doesn't mean you aren't going to be successful ... but it does mean that you come at problems from different perspectives, and that can lead to challenges and disagreements.  It can, by the way, also lead to spectacular breakthroughs, especially when each side respects the other.




P.S.:  Where do I (Kevin) stand, on this continuum?


I skew significantly toward Forecasting.  This is why my message is so different than what you hear in the Vendor community, is why my message is so different than those shared by leading analytics experts, and is why my message is not always welcome at leading conferences.


You really have to look back at the influences that shape each analyst, over time.  In my case, I was trained as a Statistician, which pushed me toward Optimization/Prediction.  Then I entered the Direct Marketing world, and noticed that Executives are not fans of Optimization/Prediction ... they tend to gravitate toward a Segmentation/Forecasting quadrant that was opposite of my training.  

In other words, I had to change my approach.


I borrowed heavily from things that were done at Fingerhut in the 1980s ... where they used a Segmentation/Forecasting bridge known as the rolling twelve-month file to demonstrate the health of the business independent of marketing campaigns.  I borrowed heavily from Jim Fulton, a former Lands' End co-worker who is now a Database Marketing consultant (Customer Metrics) ... he heavily leveraged the Forecasting system to demonstrate that new customers, not customer loyalty, lead to a healthy, growing business.  I borrowed from Lands' End, who skewed heavily toward a Segmentation/Optimization platform.  I borrowed heavily from Spiegel, a company that used a clever Segmentation/Forecasting bridge (I'll cover this in a future post) to measure file health.  I borrowed from Eddie Bauer, where pure Segmentation systems failed to improve the health of the customer file, pushing me toward channel interactions (Forecasting).  I borrowed from Nordstrom, a heavy Segmentation/Forecasting culture that "ripped the band aid off" on strategies without the need for Optimization.  I borrowed heavily from some of the early Forecasting/Prediction work done at the Santa Fe Institute in the mid 1990s, and borrowed heavily from Mathematical Biology (Forecasting/Prediction).


Notice that the word "Forecasting" keeps coming up in the activities that shaped my consulting focus.  I skewed farther and farther away from a focus on marketing campaigns over time, skewing more and more toward holistic customer migration patterns independent of marketing campaigns.


My experiences skewed me dramatically toward the Forecasting system.  I'm an enthusiastic advocate of moving beyond analysis of marketing campaigns.  I realize not many agree with me.  No worries!


Your experiences skew you to one system, or to a combination of systems.  No skew is right or wrong, all are appropriate.  You simply have to align your system with your core beliefs, and hope that your system is in alignment with those who are tasked with implementing your beliefs.