November 12, 2009

Analysts and Executives

If you are an analytics expert, and only have time to read one long blog post this year (i.e. an article longer than 140 characters), read this article from Joseph Carrabis. Jim Novo tweeted the article yesterday, thanks for pointing it out.

The article helps illustrate a huge gulf between Analytics experts and Executives, one that is about as big as the Gulf of Mexico.

Let's go back to 1994. I was Manager of Analytical Services (what would now be called 'Business Intelligence', to use the parlance of the day) at Lands' End. And I had just analyzed the results of a year long 2^7 factorial design (called MVT, or 'Multivariate Testing', to use the web analytics parlance of the day).

The results of the test were, to say the least, controversial.

One VP managed a $40,000,000 business unit that generated $2,000,000 profit, as measured by Finance. The test had 128 segments, 64 of which did not receive any catalogs from this business unit. Comparing the 64 segments that received catalogs from this business unit to the 64 segments that did not receive catalogs from this business unit yielded the following results:
  • Receive Catalogs From This Division: Total Business = $900,000,000, $8,000,000 profit.
  • No Catalogs From This Division: Total Business = $885,000,000, $8,300,000 profit.
  • Incremental Difference: $15,000,000 Demand, ($300,000) profit.

In other words, when you stop mailing catalogs from this division, customers re-allocated their demand to other catalogs. A $40,000,000 business unit that generated $2,000,000 profit was, in reality, a $15,000,000 business that was losing $300,000 profit.

The results were statistically significant, meaning that if we rolled out a strategy where we shut down this division of the company, we'd make the right decision at least 95 times in 100.

My job was to present my findings (there were similar findings for other business units) to the VP team that led each of the business units involved in the test. The results were from a perfectly executed test with statistically significant findings that no Analyst, I repeat, NO ANALYST, would question. Any Analyst would ask how quickly the new strategy could be implemented.

I stood in front of the team of Executives, including our CEO. I shared the results.

What do you think happened?

Let's just say that the audience didn't respond with enthusiasm.

Imagine being an Executive, earning $175,000 a year, having some dweeby, geeky, inexperienced, dolt-like wing-nut pimple of an employee pull out 85 powerpoint slides generated from the results of a 2^7 factorial design that suggests the Executive should shut down his division, now. Do you think the Executive will embrace the findings? Do you think the Executive will say, "Ok, that sounds great. I'll just lay off 155 employees and wrap things up before Christmas, and I'm sure I'll get another job in another city after pulling my kids out of the school they love, because your results are absolutely self-evident?" Or do you think the Executive will fight tooth and nail for his/her job, and the jobs of the people s/he leads?

Hint: The Executive will do the latter. And so would you.

What is completely missing from our real-time, web analytics, business intelligence, on demand, multi-channel integrated world of data is leadership.

Leadership doesn't lie about or create false expectations about the power of web analytics or business intelligence. You're not going to triple your conversion rate by switching from Omniture to Coremetrics. You're not going to increase comp store sales by 5% by implementing Business Objects, MicroStrategy, SPSS, or SAS. Leadership will clearly communicate what the data is, and what it means.

Leadership makes connections between Executives and Analysts. An Executive is far more likely to trust an Analyst if the Executive has worked with the Analyst, and realizes that the Analyst is actually representing "The Voice of the Customer". It is the job of the Analyst to convert geeky, dweeby, nerdy findings into a "story" that resonates with the Executive. The Analyst cannot do this unless the Analyst has been in meetings with the Executive, knows how the Executive thinks, and knows what the Executive needs to be successful. You don't ever set up an Analyst to fail by throwing the Analyst in with the wolves. You create a safe environment, so that the Analyst can share findings without having to duck a flurry of fists.

Leadership teaches, constantly. It is never about being "right". Too often, the Analyst has data that appears to be compelling. The leader teaches context, always trying to illustrate that the Analyst is there to support a greater mission, and by doing so, the Analyst will benefit. I'd rather have an Analyst with a Bachelor of Science degree providing solid findings in a team environment than a Doctorate employee demanding that the company implement highly sophisticated findings immediately, regardless of the repurcussions.

Leadership translates information into English, or the language used in the country where you work. In other words, you remove the geek-speak, the web analytics parlance or the business intelligence parlance or the statistical mumbo-jumbo that you use when talking with Analytics experts. Nobody cares that a landing page conversion rate will outperform another landing page, based on a test of 22,948 visitors with a T-score of 2.07. Every Executive cares that the test suggests that annual sales will increase by $394,000 if a certain strategy is employed, and that the Executive will make the right decision 95 times in 100.

What is missing today is Leadership. It is missing from the Vendor community, it is missing all across Executive teams that lead Corporate America, and it is sorely missing from conference agendas. We need fewer "three easy steps to Twitter success" lectures. We need more "how to communicate with an Executive in order to be effective" lectures.

If you are an Analyst, find one person, any person, in your company who appears to make magic happen through people, by working well with others. Ask to adhere yourself to this person for three months, so that you can see how this person does her job. Then emulate this person. Your company is not going to do this for you, you must take the initiative yourself.

If you are an Executive, invite an Analyst to a meeting as an observer. Have the Analyst sit in the back of the room, and don't let the Analyst speak unless spoken to. Let the Analyst hear real business issues, and let the Analyst listen to the interactions that happen as decisions are actually made. There's no more valuable thing for an Analyst than to see how things actually work, so that the Analyst can calibrate work in a way that makes the Analyst more effective.

Analysts and Executives. Leadership is the glue that could actually join these two audiences in a mutually beneficial relationship.

12 comments:

  1. Good morning,
    Thanks for the nod, thanks for this post in follow-up.
    Gosh, if only I could do that 140 character thing...
    - Joseph

    ReplyDelete
  2. Well said, Kevin!

    I would add this, speaking of leadership: your company may be organized in such a way that prevents you from taking advantage of all these analysts can bring to the table.

    Think about structural barriers to the free flow and use of performance information. For example, should your web analysts report to Finance instead of Marketing?

    That depends on whether Marketing is as open to bad news as they are to good news!

    ReplyDelete
  3. There are many times when a web analyst would benefit by reporting to Finance!

    ReplyDelete
  4. Brilliant post, Kevin. Simply brilliant. I couldn't agree more with your points about leadership and appropriate communication. I really appreciate that you're putting the burden of making this happen on both the analyst and the executive. It's absolutely appropriate for each to understand the other's perspective, language and issues. Executives need to work to better understand numbers and analysts need to work to better communicate them and understand the business context. This could be my favorite post of yours yet!

    ReplyDelete
  5. Kevin, this should be repeated often.

    Having the data is only part of the battle. Presenting the data effectively and then driving action are even more important and often times more difficult.

    ReplyDelete
  6. This sounds like a call to action to me:

    "What is missing today is Leadership. It is missing from the Vendor community, it is missing all across Executive teams that lead Corporate America, and it is sorely missing from conference agendas. We need fewer "three easy steps to Twitter success" lectures. We need more "how to communicate with an Executive in order to be effective" lectures."

    Well said Kevin.

    ReplyDelete
  7. Thank you both, Jeremy & Christopher!

    ReplyDelete
  8. Jennifer Veesenmeyer2:57 PM

    Amen.

    ReplyDelete
  9. Agreed that leadership needs to emerge. As I wrote in my original post, "...no one stepping up to lead the way...".
    For what it's worth, I'm hoping to develop some socially viable and culturally effective solutions in Part 2. I'm very open to help and suggestions in this. - Joseph

    ReplyDelete
  10. Ned Kumar1:41 PM

    Kevin, great post. I agree with you totally (and preach) about an open channel and increased cooperation between analysts and Senior Management. One point I do want to add is the art Change Management -- many in our community do not realize the importance of knowing when/how to convey a message. As you note, it does not matter if you have the most logical solution with a zillion support points and evidence - Execs will not listen to it unless they are ready. [I think] it is the role of the analytics group to not only to do analysis but also know how to socialize it with Execs and other stakeholders in the company so that the results are accepted as it should be.

    ReplyDelete
  11. Anonymous6:19 AM

    Lets hear more about this 2^7 experiment and how it drove a conclusion to close a business unit!

    ReplyDelete
  12. It did just the opposite. The test showed that several business units were not truly generating incremental demand, they were instead cannibalizing existing businesses.

    The logical outcome of the test would be to significantly modify the existing strategy. The actual outcome was to conduct business as-is, regardless of what the test results suggested.

    That's the thing with data --- it really doesn't matter what the data tells you that you should do --- what matters is how the corporate culture responds to information. And if the corporate culture doesn't like what the data suggests, it doesn't matter how good of a job you do as an analyst, the data will not be heard.

    ReplyDelete