Big Data

Maybe a quarter of the questions I get these days are about what some call "Big Data".  For a brief primer, please refer to Wikipedia's definition of Big Data (click here).

There are two distinctly unique aspects to what the pundits call "Big Data".

• Technology.
• Applications.

I won't focus on Technology.  Rest assured that large vendors will develop slutions that promise to save the world.  You'll purchase the solutions, and you'll achieve varied levels of success ... just like you've been doing since we moved from mainframes to PCs in the late 80s.

I focus on Applications.

There are at least four key Application concepts to pay attention to.  They are:

1. Complex Adaptive Systems.
2. Dirty Algorithms.
3. Hyper-Optimization.
4. Brand Interaction.

Complex Adaptive Systems (click here):  This is what we fail to understand about our world.  Things connect, and they interact with each other, often yielding unpredictable outcomes.  In the catalog world, cataloger interaction with co-ops is representative of a Complex Adaptive System.  Catalogers volunteered customers to the co-ops, co-ops used algorithms to redefine the names, and then resold the names back to catalogers.  There are many participants in this system, dependent upon each other.  Their interactions yield unpredictable and unusual results (i.e. co-ops spinning 55+ customers to catalogers, accelerating the evolution of catalogers).

Dirty Algorithms:  This is my term, and it will be the bane of our existence!  Dirty algorithms seek to maximize the profitability of a portion of a Complex Adaptive System, without understanding how the Dirty Algorithm soils the entire Complex Adaptive System.  Example?  Easy!  Credit Default Swaps and their role in the meltdown of the global economy in 2008.  When a financial institution buys insurance to "spread the risk" of an investment, the financial institution is inserting a Dirty Algorithm into the Complex Adaptive System.  In the "Big Data" world, companies will routinely insert Dirty Algorithms into Complex Adaptive Systems.  9 times out of 10, this will not be done with malice, but rather, ignorance of how Complex Adaptive Systems work.  1 time in 10, this will be an act of pure evil.  We won't know the difference, we'll just be cleaning up messes all the time.

Hyper-Optimization:  We're at least a decade in to the era of Hyper-Optimization, and thus far, the results have not been pretty.  The best example of Hyper-Optimization happens in web analytics - earnest, honest, and well-intentioned analysts seek to increase conversion rates.  They take friction out of the system, spending time, resources, and money improving conversion rates, not realizing that the actual behavior exhibited by customers does not change ... that in reality, the web analyst caused a customer who visited the website 4 times before a purchase to visit 3 times before a purchase.  When the underlying behavior does not change, we are Hyper-Optimizing ... changing an outcome that does not fundamentally change the behavior.  This happens when we measure the wrong attribute.  If the web analyst measured annual frequency and annual repurchase rates, the web analyst would not Hyper-Optimize a meaningless outcome.  Email subject lines also fall under Hyper-Optimization ... here, marketers realize that conversion rates won't increase unless 20% off plus free shipping offers are provided.  The problem in this form of Hyper-Optimization is that the email marketer only attracts discount buyers, further fueling the need for future discounts.  If this behavior continues, the email marketer is no longer engaged in Hyper-Optimization, but rather, has introduced a Dirty Algorithm into the Complex Adaptive System.  Along these lines, Cyber Monday is the most disappointing version of Hyper-Optimization, whereby online brands now offer 30% off plus free shipping to yield the best final Monday of November in history, never minding that sales are depressed in the three weeks prior to Cyber Monday to wait for the discount.  Hyper-Optimization is a direct outcome of terrible measurement practices.

Brand Interaction:  Here's where most of us enter into our relationship with Big Data.  Most of us will treat Big Data as a glorified form of Campaign Management.  In Campaign Management, actions were linear and additive.  We mail 100,000 catalogs, we get $500,000 in demand ... we send 1,000,000 email messages, we get $200,000 in demand ... we buy 10,000 clicks for $0.50 each and we get 300 orders ... Cause and effect.  This is the world most of us honed our marketing skills in, in the 1980s and 1990s, a pre-Google world.  The reality is that we've always operated in a Complex Adaptive Ecosystem (called "the economy"), but we didn't have the data to help us understand the truth.  Most of the Big Data hucksters will operate in this realm, promising real-time decisions that dramatically boost profitability.  What they'll be doing, however, is a simple transfer of demand, from one party to another.  Yes, on a macro-economic level, growth can happen.  But by and large, on the level we deal with, we're trading demand among players.  Big Data solutions providers will simply push demand back and forth between those buying (or not buying) solutions ... and in some cases, will, by accident, interject a Dirty Algorithm that will cause all sorts of problems, or will Hyper-Optimize (pushing demand out of certain windows, into others).

Your Job?  Be smart, and I mean that with all honesty.  Most Big Data solutions will sound very seductive, on a Campaign Management / Brand Interaction level.  Your job is to ask solid questions, as you try to understand how a Big Data solution interacts on a Complex Adaptive System basis.  Are you being sold a Dirty Algorithm?  Who is demand being transferred from?  Are you simply Hyper-Optimizing a situation without yielding long-term growth?  As an example, remember that every time you use Dictionary.com, more than 200 cookies are placed on your computer.  Your simple level of inquisition at Dictionary.com results in hundreds of businesses harvesting information, pushing your inferences into the Complex Adaptive System called "Marketing", with Dictionary.com obtaining profit.  Those companies will attempt to influence you via Brand Interaction, in the form of Campaign Management.  You need to learn how this impacts you as a customer, and how it impacts the company you work for.

Go beyond the hype.  Study Big Data within the context of Complex Adaptive Systems, Dirty Algorithms, Hyper-Optimization, and Brand Interactions.  You'll find that Big Data is far more interesting at this level than what you read about in trade journals.