Web Analytics, Online Marketing, Metrics

Note: Every Monday and Wednesday in August, we'll be talking about Web Analytics and Online Marketing Simulations (OMS). The discussions will build in depth and math as the month moves along.

There's never been a better time to be a Web Analytics expert!

There's never been a more challenging time to be a Web Analytics expert, either.

Do you want to know how customers who visit your site via iPhones convert? No problem! Care to measure the bounce rate of new customers? Have at it! Need to know the cost per conversion of visitors from Wyoming visiting because of PPC? Easy!

Online marketing is calibrated around a metric called "conversion rate". We are able to segment visitors via a veritable plethora of dimensions, using a mouse to drop in metrics based on dimensions we define on the fly. It's easy, it's fun, it is actionable, it increases sales! More important, we're able to create A/B tests, allowing us to optimize our results in real time. Technically, we know more about how customers interact with advertising than ever before. What's not to like about being a Web Analytics expert?

That's one side of the spectrum.

Over on the other side of the spectrum are what I call "strategic questions". Strategic questions are a different animal altogether. Strategic questions are harder to answer, because the answer isn't found by using metrics to optimize conversion rate.

Strategic Question #1: You are a retailer with three stores in Akron, OH. Management is considering closing one store, and is hoping that the other two stores and your e-commerce website will pick up the sales lost if one store is closed. What is the impact on website sales if one store is closed?

Strategic Question #2: Your merchandising team added a new product line in July. This product line is already responsible for 5% of company sales, a huge success. However, the product manager for an existing product line experienced a 25% reduction in sales in July. She believes that the new product line cannibalized her assortment. Other product managers don't agree, because their products experienced sales gains during July. Nobody tested offering/not-offering the new product line to customers. What impact did the new product line have on the old product line?

Strategic Question #3: You implemented five new initiatives. Each initiative increased conversion rate, based on A/B tests, by 10%. And yet, in total, your conversion rate is down 10% vs. last year. Your CEO holds you accountable for increasing conversion rate. How are you going to demonstrate that the conversion rate decrease is not your fault? What offline data do you need to make your case to management?

Strategic Question #4: Management is considering closing down your catalog division. Management wants to know what e-commerce sales will look like in 2015 if there has not been a catalog to support e-commerce sales for a five year period of time. What is your estimate for e-commerce sales in 2015, without a catalog division there to support e-commerce sales?

Here's the trap that the Web Analytics expert is in. All of the big Web Analytics providers (Coremetrics, Unica, Omniture, for example) can help you answer these questions. You can, in theory, import data or link to data or export data or build a data mart and, technically, get to an acceptable answer.

But if your focus is on the powerful combination of conversion rates and optimization, it will be hard to conceive an analytical framework that yields an answer acceptable to a CEO.

As mentioned a few weeks ago, we're going to spend a lot of time in August exploring how our focus on conversion/optimization limits our ability to answer strategic questions. In August, we'll show how we can use a simulation environment to better understand strategic issues in online marketing. We'll explore how we can see the future via a different framework.

If you want to prepare for our month-long discussion, consider these authors, folks who use different metrics to re-define their craft, or explain how the "new" is really borrowed from the "old".

• Basketball: The Wages Of Wins, explaining why scoring does not lead to wins.
• Football: Advanced NFL Stats, illustrating the ways that traditional metrics fail to explain success.
• College Football: Smart Football explains why the "spread" offense is not orginal, is largely borrowed from plays from fifty or more years ago.
• Here's an article about Patrick Ewing that has many similarities to Web Analytics, arguing that you optimize the end result by sub-optimizing components, by having less talented players shoot more so that the best player's performance is optimized. Of course, this is contrary to Web Analytics and Online Marketing theory, but it does help explain why a decade of optimization sometimes leads us to lower conversion rates over time.