Fetzer's Footwear: Testing

Today, I'm meeting Lauren Fetzer at Willow Harbor, one of two ports on Madrona Island. Ms. Fetzer is standing on a dock, iPod Touch in one hand, Red Bull in the other hand, earbuds in her ears.

Kevin: "What are you listening to?"

Lauren (pulling earbuds from her ears): "DC Talk, Between You And Me". This is my church, being outside on Madrona Island, listening to DC Talk.

Kevin: "Amen".

Lauren: "Let's walk the docks. Six years ago, I see this giant yacht, and then Johnny Carson zooms by on a motorized scooter, heading to the yacht. This yacht had to be three stories tall, bigger than half of the cruise ships you see go through the strait. So you never know who you are going to see here."

Kevin: "What are we talking about today?"

Lauren: "Testing. We do a lot of testing at Fetzer's Footwear. We work with a service called Pinnacle Testing. They use Java Script to allow us to test different offers or creative strategies on our home page and various landing pages."

Kevin: "How many tests a year are you running?"

Lauren: "We run two tests a month, and this gets expensive. Pinnacle Testing charges $6,500 a test, so we spend more than a hundred thousand dollars a year executing various tests."

Kevin: "What have you learned?"

Lauren: "Maybe the most important thing we learned is that much of what we learn has a half-life. Certainly, there are eternal truths. Best customers visit our site to see what is new. New customers have higher conversion rates on our best selling items. That contradiction alone was worth the cost of testing, we basically serve up different pages to new and existing customers, featuring existing product to new customers, featuring new product to existing customers."

Kevin: "What do you mean by the term half-life?"

Lauren: "Most of what we learn has a half-life. We'll run a test to see where the best place is to put our shopping cart on the home page. We learned upper-right was the best place, that customers converted 18% better. Then six months later, we ran the same test, and found that customers converted 6% better. Then last month, we learned that the upper-left corner of the home page worked 9% better."

Kevin: "Is that due to sampling error?"

Lauren: "The folks who run the tests for us at Pinnacle Testing tell us they use statistical significance tests, so they promise that the results are statistically valid."

Kevin: "The tests are valid based on a series of assumptions. When you execute a test like this, you assume that the audience visiting the website in the future will be identical to the audience visiting the site during the test period. Does that assumption hold in the tests you run across time?"

Lauren: "Certainly not. The audience, in fact, would always be different. In December, the composition of the audience is totally different than the composition of the audience in May."

Kevin: "That means you are violating the assumptions of each test, if you assume that the results of a test in May will hold in November. And when you violate the assumptions of a test, you make sure that your results cannot be replicated, no matter what folks tell you about the statistical significance of a test. It's a big reason why all of these tests sound scientific, but in reality, can often represent a point-in-time outcome that isn't easily replicated. It's really a misinterpretation of statistical testing, we violate the assumptions underlying the original test."

Lauren: "Help me understand this half-life concept. So often, we run these tests, and we see that one creative treatment will outperform another by 65%. We get huge conversion rate increases for a month or two, but then the increases seem to evaporate. We test a big product swatch, and that works great, and then six months later, we re-test and find that the small product swatch works better. What's up with that?"

Kevin: "There's two reasons for that. First, customers get bored. You dazzle them, then they get used to what you did, and they revert back to their old behavior. Think about how spectacular e-commerce is today, compared with ten years ago. And yet, conversion rates over time have steadily decreased. Customers almost have to be 'shocked', if you will, in order to see increased productivity, and you cannot continually shock customers into improved performance on a consistent basis. So in some sense, all of this conversion optimization is fool's gold, if you will, because you seldom if ever truly improve the annual retention rate. Well, fool's gold is a harsh term. Simply put, the outcomes of the tests evaporate in performance over time, they have a short half-life."

Lauren: "So am I wasting my $100,000ish annual investment in testing?"

Kevin: "Absolutely not, in fact, you're way ahead of your competition. The important thing is to focus tests on things that matter. Font sizes, product swatch sizes, colors of buttons, size of buttons, subject line testing, all of that stuff has a short half-life, it is transient, and quite honestly, none of it addresses the real issue with a business."

Lauren: "And the real issue is?"

Kevin: "The real issue is the merchandise you sell and the story you tell. What folks test online is so similar to testing how you display the mannequins in a retail store, or the color of the sale signs in a retail store. Sure, that plays a role in causing a customer to purchase. What really plays a role is the merchandise you sell or the story you tell. The only reason a customer is shopping your brand is because you sell merchandise the customer wants. All of the other testing ideas skirt the real issue of presenting great merchandise in a way that resonates with the customer, or skirts the real issue of writing copy or blogs that romance the customer or educate the customer in a way that causes the customer to feel compelled to shop with you. So few people talk about this stuff, but this is the stuff that truly matters. I mean, honestly, do you care if McDonalds changes the color of the arches? Would you not go through the drive through if you were in a pinch and McDonalds was just down the road and then you see that they changed the color of the arches to an off-yellow color?? That's the kind of stuff folks want you to test on your website. It is fool's gold. It has a short half-life, and it diverts you away from true business strategy. Honestly, I think we are lazy. It is easy to test a small green button vs. a large green button vs. a small red button vs. a large red button, it takes no thought, no creativity, no imagination, it is psuedo-strategy. It is really hard to create four different stories surrounding the reason why a customer should purchase a blu-ray DVD player, then test how the four different stories resonate with the customer. Give the latter a try!"

Lauren: "So when we learned that new customers prefer existing product, and existing customers prefer new product, we learned something timeless?"

Kevin: "Quite possibly. Try testing strategies that are 'timeless', if you will, or at least test ideas that have longer half-lives."

Lauren: "Why don't we ever read about this style of testing? When you read the blogs, all you ever see are articles about testing tactics on the home page or on landing pages?"

Kevin: "Nobody is going to tell you the real secrets, right? The most delicious, most interesting findings are never discussed publicly. Pinnacle Testing knows this, they can't ever share the real findings with anybody, those findings are proprietary to the brands they work with. It certainly isn't the fault of the testing organization."

Lauren: "I suppose you are right. We never tell anybody what we learn about the merchandise testing we do. We view that information as being proprietary."

Kevin: "So keep testing. Use all that fancy multivariate testing and learn whatever you can possibly learn. Everything you learn puts you one step ahead of competitors who choose to never do testing. But test things that matter. Most of the time, the stuff that matters is what you offer the customer on the home page or on a landing page, or what matters is how you romance the customer. That's the secret sauce!"

Lauren: "Ok, then. Hey, you want to stop at the sundry shop and pick up a Red Bull for the road?"

Kevin: "No."

Lauren: "Do you ever drink Red Bull?"

Kevin: "No."

Lauren: "Would you drink Red Bull if they changed the color of the can, or the size of the can, or if they personalized the can in any way for you?"

Kevin: "No."

Lauren: "I see. So it's the merchandise that matters, it is what is inside the can that matters, huh?"

Kevin: "Amen!"