Showing posts with label Analytics Thursday. Show all posts
Showing posts with label Analytics Thursday. Show all posts

January 19, 2011

Analytics Thursday: Listening, Timing, and Permission

Old School:  It's 1994, and I'm working at Lands' End as Manager of Analytical Services.  For the past year, I helped execute and analyze results of a test to measure the impact of every business unit on each other.  We just finished executing the test ... fifty-two weeks of a 2^7 factorial design that yielded 128 test segments and countless tidbits of information on how customers shopped.

I expected to complete the results of the analysis in a couple of months ... after all, I had a day job to do, so I set an arbitrary analysis completion date.  Nobody in Management complained about the deadline I set, so I believed that all was good.

And then I'm listening to our business head tell an individual that, in seven days (the following Thursday), she was meeting with all business heads to discuss how to strategically approach the marketing of each business unit in 1995 and 1996.

Three thoughts came to mind.
  1. I wasn't invited to the meeting (nor should I have been, I was a lowly Manager).
  2. My data wasn't going to be shared at that meeting.
  3. The entire reason we executed the year-long test was because we wanted to determine how to strategically approach the marketing of each business unit in 1995 and 1996.
What to do ... what to do ... what to do?


I spent the entire weekend compiling the results of the test.  I ignored every geeky metric that existed ... you know, stuff like significance tests and p-values and confidence intervals and all that stuff.  I pretended that I was a business head ... what did this person what to learn, or more important, what did this person need to learn?

I met with an Executive in the Finance department ... I played basketball with this person three times a week, so I felt comfortable asking this person for detailed financial information about each business unit.  After sharing my presentation with him, he showed me how to calculate a full profit and loss statement for each business unit ... with catalog mailings or without, after accounting for all fixed and variable costs.  This was a really important step, because profit data told a story that couldn't be easily inferred with standard KPIs.  The story that I needed to tell could not be properly told or understood without profitability data.


I ended up with eighty (80) Powerpoint slides.  

I still didn't have a forum for sharing the results.


I hastily arranged a meeting for the following Wednesday (the day before the meeting with the heads of each business unit), inviting my department head, the other Managers in my department, and the folks who helped me execute the test.


On Wednesday, I printed color copies of the presentation (color copies were a big deal in 1994), handing out eighty pages of hard work to each Manager.  Our meeting was at 2:00pm.  Our department head didn't show up ... and that's a bad thing, because this person needed to see the results, because she was the only person invited to meet with the department heads.


I started the meeting at 2:00pm, regardless of her lack of attendance.  At 2:50pm, as I was wrapping up slide #74, our department head entered the room.  As I finished presenting the final six slides, our department head fast-forwarded through the content she missed.  I ended my presentation.  Our department head told our team that the information in the test was exactly what she needed to share with all department heads, and that, amazingly, there was a meeting the very next day with all department heads to go over strategy for 1995-1996.  What a coincidence!  She asked me to attend the meeting, and asked me to present the slides to the CEO and all business heads.


I accepted her invitation.  The next day, I presented the findings of the year-long test to a room split by the results of the tests.  Needless to say, the discussion was spirited, an eye-opening experience for a twenty-nine year old who had not previously had an opportunity to participate in a meeting of this magnitude at a billion-dollar a year company.




Modern Application:  Give Twitter a read, and you'll see analytics experts providing tips and techniques for success.  We're taught how to execute a test, how to measure significance, how to avoid pie charts, how to create a dashboard that outlines the findings.


We're almost never told about listening, timing, and permission.


First, keep your ear to the ground.  This isn't about gossip, this is about taking the temperature of your organization.  What are people wondering about?  Do you have answers for the questions people have?  Do you have a forum for providing the answers?  I can't tell you how important it is to listen.  Listen to Executives.  Listen to Directors.  Listen to Customers.  Listen to the CEO.  Listen to the CEO's Administrative Assistant.  Practice taking action, based on what you hear.  You're not going to be perfect.  I tried the exact same approach in 1997 at Eddie Bauer ... and it was a disaster.  There aren't "Six Easy Steps To Getting A Seat At The Executive Table".  You try things, you learn, you adapt.  View it like a free throw shooting percentage in basketball ... shoot 60% this year, then 70% next year, then realize it's going to be very hard to ever shoot 75% or better.


Second, be savvy ... timing is everything.  If I complete the work just before the approved deadline, it's entirely possible that the work never sees the light of day.  Know when your Executive team has scheduled meetings.  If you don't have a relationship with your Executive team, initiate a relationship with the Admin who supports the Executive you want to work with.  The Admin knows when important meetings are scheduled, the Admin knows when weekly meetings are held and who attends those meetings.  Hand-deliver your results to the Admin ... don't rely on Executives to visit your fancy intranet site, don't expect an Executive to review your important, KPI-loaded customer analysis dashboard.  Take matters into your own hands.  Tailor your projects around key Executive meetings.  At Eddie Bauer, in 1997, there was this thing called a "QPM", or "Quarterly Planning Meeting".  Once a quarter, all Executives got together to discuss the performance of the business.  Not surprisingly, I tailored delivery of all important analytics projects within one week of this meeting.  Timing is everything ... you are marketing yourself, you are defining your "target audience" and you are marketing to your target audience when the target audience is most receptive.


Third, permission is flexible.  I know that's a hard concept to accept, but permission is truly flexible.  I didn't have permission to obtain company financials, heck, those numbers were never shared with anybody ... but I had data that the Finance Executive loved, I gave the Finance Executive first crack at seeing the results, and I had a prior relationship with this individual, so I obtained information critical to the story I was telling.  Furthermore, I did not have permission to attend an Executive meeting, so I had to create a forum that would grant me the permission I sought.  

We're taught to use this approach:
  • Use best practices to create a spectacular analysis.
  • Publish your findings via a dashboard, utilizing KPIs that measure ROI.
  • When Executives don't accept your findings, beat the Executives up for being Luddites.
I suggest a different approach.
  • Listening.
  • Timing.
  • Permission.
Your thoughts?

January 12, 2011

Analytics Thursday: Channel Shift to Mobile

Old School Issue:  In the Fall of 2001, the internet was in the early stages of obliterating traditional shopping habits.  The shift in customer behavior was not well understood.  Too often, the "best" customers were ones making the shift from offline to online, clouding the issue, making it look like channels add up to increased customer value.  I was working at Nordstrom at the time, as Vice President of Direct Marketing, suggesting to folks that the internet was about to fundamentally change our lives.  Our Divisional President liked to see data presented in a specific, calculated manner, before accepting the hypothesis that the internet would dramatically change customer behavior.

He asked my team to prepare the following query.
  • Identify customers who, as of 12/1/2000, were identical in customer quality ... having spent $450 - $550 in the past five years, having purchased one time in 2000, and having spent all of their previous money in the telephone/catalog channel.
  • During December 2000, require that this audience purchased exactly one time.
  • Split the audience by those who purchased in December 2000 via telephone/catalog, and those who purchased online after receiving a catalog.
  • Measure spend from 1/1/2001 to 11/30/2001 by physical channel (telephone, online).
The theory was that customers had identical historical behavior, so the only thing that was different was the choice of one segment of customers to buy online vs. the choice of another segment of customers to buy via the telephone.  Here's the results of the query.
  • Telephone/Catalog Buyers Future Spend, Telephone Channel = $100.
  • Telephone/Catalog Buyers Future Spend, Online Channel = $10.
  • Telephone/Catalog Buyers Future Spend, Total = $110.
  • Telephone/Catalog Buyers Future Profit = $18.
  • Online/Catalog Buyers Future Spend, Telephone Channel = $45.
  • Online/Catalog Buyers Future Spend, Online Channel = $55.
  • Online/Catalog Buyers Future Spend, Total = $100.
  • Online/Catalog Buyers Future Profit = $15.
What Did We Learn?
  • Channel Shift was a real phenomenon.  Once the customer shifted to a new channel, subsequent behavior was permanently altered, with existing channels losing share to new channels.
  • Channel Shift resulted in a reduction in future profit.
  • The results of the query weren't popular, but they were real.  Channel shift was going to require a fundamental shift in how to manage customer behavior.  Folks didn't like the results, folks didn't believe the results.  It didn't matter ... the customer clearly illustrated a change in behavior.
Modern Application:  The same methodology applies in the shift from the browser to a mobile app.  The online generation is going to suggest that the online experience fuels the mobile experience.  A new generation of business leaders are suggesting that the mobile experience is fundamentally different, altering customer behavior forever.  Use the methodology outlined in this article to accurately measure channel shift from the online browser to the mobile app.  I'm guessing that the results will be similar to the shift from offline to online a decade earlier.

January 05, 2011

Analytics Thursday: Guesstimates

For awhile here, we're going to talk about old-school issues that have relevance in our modern digital world.  The series will be called "Analytics Thursday".

Old School:  Today, we go back to 1988.  I was a Statistical Analyst at the Garst Seed Company, in picturesque Slater, Iowa.  My job was to analyze corn and sorghum hybrid experiments.  Sometimes, our experiments yielded odd results.  Once, a deer "romped" through our experiment, ruining the results of a test.  Let's assume that we were analyzing four hybrids, and there were two sections in the field where we were testing.  Here are the results of the test:
  • East Side of Field, Hybrid #1 = 120 bushels.
  • East Side of Field, Hybrid #2 = 110 bushels.
  • East Side of Field, Hybrid #3 = Ruined by Deer.
  • East Side of Field, Hybrid #4 = 100 bushels.
  • West Side of Field, Hybrid #1 = 140 bushels.
  • West Side of Field, Hybrid #2 = Ruined by Deer.
  • West Side of Field, Hybrid #3 = 130 bushels.
  • West Side of Field, Hybrid #4 = 110 bushels.
My job was to determine the hybrid that had the best yield, in bushels.  We approached this by using a regression methodology ... three dummy variables for Hybrid #1, Hybrid #2, and Hybrid #3 (all compared against our base, which is Hybrid #4), and one dummy variable for the East Side of the Field (compared to the West Side of the Field).  We predict Yield (in bushels) as a function of our dummy variables:
  • Equation = 112.5 + 25.0*(Hybrid_1) + 12.5*(Hybrid_2) + 17.5*(Hybrid_3) + 0.0*(Hybrid_4) - 15.0*(East_Side_of_Field).
With this equation, I can predict the yield in instances where our test was ruined by deer.
  • East Side of Field, Hybrid #1 = 120 bushels.
  • East Side of Field, Hybrid #2 = 110 bushels.
  • East Side of Field, Hybrid #3 = 115 predicted bushels.
  • East Side of Field, Hybrid #4 = 100 bushels.
  • West Side of Field, Hybrid #1 = 140 bushels.
  • West Side of Field, Hybrid #2 = 125 predicted bushels.
  • West Side of Field, Hybrid #3 = 130 bushels.
  • West Side of Field, Hybrid #4 = 110 bushels.
Now that we've corrected for each "bad data point", we can create averages for each hybrid (a methodology called "least square means").
  • Hybrid #1 = 130.0 bushels.
  • Hybrid #2 = 117.5 bushels.
  • Hybrid #3 = 122.5 bushels.
  • Hybrid #4 = 105.0 bushels.
We filled in the holes in, providing guesstimates where deer ruined the test.




Modern Application:  If you are a Savvy Web Analyst, you can use this methodology to predict what might happen if you don't have a promotion on a certain day.  East/West portions of the field are like having a free shipping promotion, or not having a free shipping promotion.  Each hybrid is a day in your fiscal year.  Use "Conversion Rate" instead of "Year".  You'll have days where you ran goofy promotions, those are the days that are like deer ruining our hybrid experiments.  This methodology allows you to make an accurate guesstimate as to what might have happened if you ran normal promotions, or no promotion at all.