August 31, 2010

Analytics Fail

If you follow me on Twitter, you already know about my mistrust of services like Groupon ... no, not a mistrust of Groupon and their business model, but rather, a mistrust of the marketers who see a one-day spike in promotional sales and feel like they've grown their brand in a healthy manner.

The failure, of course, is the analytics used to measure promotions.

You see, more than 95% of what we analyze surrounds a day, a week, a quarter, a season, a year, an item, a department, a division, or a campaign. Marketers, in particular, want to know how a promotion worked.

"Worked" is a wonky word.

Take the Chicago Bagel Authority (thanks @choltan!).

Most analytics tools measure if we sold a lot of merchandise on any given day. It's easy to find a report that says we sold $x on Monday.

It's not easy to find a report that says we generated $y profit on Monday, is it? Seriously, go ask any person in your company if there is a report that tells how much profit was generated on Monday.

Seriously. Go ask somebody.

Last week, I asked 1,970+ followers on Twitter to tell me if they measure the percentage of customers who purchased last year and then purchase again this year. It turns out that this is one of the most important metrics you can measure.

Not one individual said they measure customers in this way.

Every day, we experience an analytics fail. We measure what is easy. It's easy to measure sales, so we spend our time optimizing sales growth. It's hard to measure profit, so we generally ignore it.

In Web Analytics, we've trained a generation of analysts to look only at what Google tells us to look at. Did you ever stop and think about that, the ramifications of a generation of individuals who look at what Google tells them to look at, a generation of individuals who haven't written a line of code, code that allows them to investigate and answer their own questions?

Back to Twitter. When I expressed my frustration with services like Groupon gouging retailers (like Gap) of all of their gross margin, the analytics and marketing community fired back with every conceivable marketing theory.
  • "They will make up the gross margin dollars in repeat business."
  • "Not all Groupons are redeemed."
  • "The promotion increases the number of new customers, and don't you always talk about the importance of new customers?"
  • "You obviously don't think like a retailer, it's about getting butts in the store."
  • "Gap will make money by increasing comp store sales which increases market share which drives up the price of stock which increases shareholder value."
When I asked our community on Twitter to provide the customer metrics that prove their theory, Twitter went silent.

Analytics Fail.

We simply don't measure what matters. We almost never look to see if the customers who purchased via a Groupon promotion ever decide to repurchase. We almost never look to see if we generate profit when we create spiffy marketing promotions ... and if we are one of the rare few who measure short-term profit, we seldom follow-up with measurement of long-term profit.

We act so smart, and so advanced with our trendy, real-time analytics.

And in doing so, we damage the businesses we work for.


  1. So true! And certainly a consequence of the perpetual journey for immediate gratification. Be it the marketer who gets cheered up by a specific Google AdWord campaign "success"... success obviously being defined by the number of clicks leading to the site (*sic!*), the CEO surviving another quarter because revenues were up by 5% or the stakeholder seeing his stock going up a notch...

    When you say "we've trained a generation of analysts to look only at what Google tells us to look at." it reminds me of a post I did two years ago where I raised my concern for the "dominance" of Google Analytics. It is still very much relevant!

    I'm coming to the conclusion that the demise of the so called "web analyst" would be a good thing! An analyst is an analyst is an analyst! Get the data, gain understanding of the business, and define the problem statement (or opportunity), analyze the data & the context, and make optimal and realistic recommendations... I'm sorry we've created a class of analysts with horse blinders that limits them to the online channel and online data without any clues of the surrounding business context.


  2. John Magee7:05 AM

    I've been a follower of your writing for quite a while; I wish I could participate in more of the discussions, but I work in banking, and before that pretty much strictly in customer acquisition across a variety of businesses. Totally different challenges to your retail focus.

    I'm a big fan. Most of what you say rings very true. I do take issue with a couple of points here.

    First, to Stephane above, I would say that there is a very clear role for analysts with digital channel expertise. I have worked with a long list of very smart folks who have no clue about the mechanics of digital marketing, and always seemed poorly equipped to truly understand them quickly. I'm talking about ad serving, cookies, etc. etc. "An analyst is an analyst" is simply not true, and creates counterproductive perceptions for those who have to work with analytics teams. "A truly great analyst is infinitely versatile" is closer to the truth, but the world doesn't quite work that way.

    Secondly, I would say that this stuff isn't Google's fault. I don't think Google makes that many promises, especially in analytics. The problem is with a) bad hiring of incurious people and b) the "magic bullet" approach of software platform marketers overall - something Google is less guilty of on average, I believe.

    If you're a manager in analytics, learn how to ask tough questions of your candidates, and always press people for more curiousity in their work.

    And, like Kevin implies, never hire anyone who can't write code.

  3. Great post, definitely some interesting thoughts. I agree with the message: analysts need to be curious, dig into data, ask the right business questions (profit is definitely one! Revenue is useless without profit for context) and not just pull out whatever Google Analytics or Omniture or WebTrends etc tracks and claim that those are insights.

    I was one of the people who saw your tweet, and did not respond. I'd love to hear your take on how you would tackle this particular challenge:

    1. Content site providing buying/selling advice
    2. Related to large purchases (the type people only make every 3-10 years)
    3. No real sign-in option (so no user accounts to monitor over time - the only option for tracking visitors across a long timeframe is cookies ... eek! Who HASN'T changed computers or cleared their cookies in five years?)

    I agree with the fundamental point that sites need to look at long-term, but in some cases this is more challenging that others. In our case, we track return visitation, but understand it's directional and subject to the usual cookie data capture issues.

    So for this particular problem: I happen to be of the opinion that there is a need for such a business to have a prominent, valuable sign-up option (such that a visitor would actually choose to create an account) because I do believe long-term customer retention is important to track. However, the reality is that without that, it is challenging to look at long-term customers, even if you (the analyst) think it's vital that the company does.

    Hey, I think the crux of my comment is that I agree and would love to get more support for this view within my own org!

    And just a little PS. Much as I wish profit were a driving motivator behind analysis, keep in mind private companies aren't necessarily so forthcoming, even within the org. Again, constraints of reality.

    I believe analysts need to understand the value here, and champion for it internally, but I also understand that sometimes reality doesn't allow for these things to change quickly. I truly wish it did.

  4. John Magee Good point - and I have to admit you are right about being infinitely versatile. My rant was more about people who might be really good a cookies, JavaScript, HTML and using Google Analytics but don't have any clue about the business side of things.

    And of course, Google never claimed to mold analysts in a certain way... I guess it's just part of the natural evolution of the industry where so many new web analysts are only exposed to what Google has to offer...

    Great conversation! :)


  5. Stephane, thanks for the comment & tweets. Web Analysts are analysts, and increasingly, at the companies I consult with, management is expecting the web analyst to provide the same services as are provided by the classic SAS Programmer or Business Objects or MicroStrategy analyst or the data miner. Times certainly are changing.

    John, yours is an argument I hear every day. I had an exchange with an individual yesterday, one who criticized me and then mentioned how unique digital marketing experts are, suggesting that offline folks couldn't possibly catch up to the expertise of the digital marketing expert.

    I recently visited a client where the digital marketing director suggested that offline folks couldn't possibly understand what they do, then chose not to execute any of the very actionable strategies offered by the offline expert. I got an earful from the offline expert about the arrogance of the digital marketing expert. I got an earful from the digital marketing expert about how stupid the offline marketing staffer was.

    Both sides, of course, are right. But the longer the digital marketing expert perceives that s/he is special and different, that s/he has skills that offline folks couldn't hope to acquire, the more separated the digital marketing expert is from the rest of the organization. I see this pattern happening across my client base, and it doesn't bode well for the digital marketing expert, regardless of how brilliant the digital marketing expert is.

    Michele, I have no problem with a business model that doesn't lend itself to the analytics that I support in e-commerce and retailing, you simply do the best with what you have.

    My guess is that there is somebody in finance that is willing to share with you how a profit and loss statement is created. Once you have that, you can create your own metrics that tie to the profit and loss statement. I watched my manager walk me through this process at Lands' End in 1991, quickly learning that the finance department is the marketer's best friend!

  6. John Magee9:48 AM

    Kevin -

    I've seen your story re: the digital vs. the offline expert play out many times myself. I personally have quite a bit of experience in both. Which is to say that I didn't mean to imply that it's impossible for offline folks to learn and catch up, though I went more in the reverse direction. But it is misleading and arrogant to believe that there isn't some very important specialized technical knowledge in the digital domain. Just as it is misleading and arrogant for digital marketers to scoff at offline marketing for its lack of click-and impression-level "accountability" (quotes intentional).

    There is nothing more frustrating than a supposedly data-driven marketer who believes in only a subset of the data.

  7. There you go, good job!

  8. Kevin -
    I agree that what you describe is the norm. However I feel as more companies adapt to SaaS models with Monthly Recurring Revenue models, the focus on sales takes into account profitability or other such important goals. e.g. I need to make x sales and keep churn below y don't measure profitability directly but directly impact it. At least when I was at HubSpot we closely monitored marketing by activity (such as blogging) and its direct impact on leads, opportunity and sales, and also looked at churn.

  9. Thank you, good feedback!

  10. Kevin, I'd definitely be all over that, I'd buy many a lunches for the finance person who helps me with that. However, not all companies are willing to share the data you'd need to consider profit and loss, and that's often a corporate policy. One day though! Just need to sell the benefits.

  11. Almost any finance person at any company will tell you the "flow-through rate" ... the rate that sales are converted to profit. If you have that, then you multiply sales by the flow-through rate (say $10,000 * 0.30 = $3,000), then subtract marketing cost ($3,500), and you have a good proxy for profit ... in this case a loss of $500.

    If nobody in finance will produce that for you, hmmmmmmmmmmm, it's perfectly harmless to have that information, in fact, it's good for the company, especially if you show finance folks how marketing activities perform.

    I don't think I've ever had one company say no to giving me that information.


Note: Only a member of this blog may post a comment.

Good Vendor Employees Are Working All Around You

So I'm on a Zoom yesterday, and the individual representing the vendor did SUCH a good job. What does doing a good job look like? Patien...