Attribution experts express confidence in their ability to parse an order across the activities that caused the order.Oh boy.
I was about to purchase a new television, a 46 inch model. I decided to purchase the television from Crutchfield, because I knew I would get free shipping.
I made this decision on or around May 1.
In the first week of May, I received a small page count catalog from Crutchfield. I looked through the catalog. My television was not, to my knowledge, featured in the catalog.
In the second week of May, I researched the television I wanted to purchase. I used Google. I visited Amazon. I visited J&R Music World. I visited Best Buy. I found the television I wanted to purchase. I compared prices across brands. I then visited Crutchfield. The television I wanted was available. I placed the item in my shopping cart.
In the third week of May, Crutchfield sent me an e-mail marketing message ... and surprise surprise, the television I wanted was featured in the e-mail marketing message. Who knew?! It's almost like Crutchfield scanned my cart and then advertised the item to me.
Late in the third week of May, Crutchfield lowered the price of the television. Good for me!
Early in the fourth week of May, while reviewing my shopping cart, Crutchfield made me an offer. They offered to give me a soundbar/subwoofer system, valued at over $300, for free, if I purchased my television. Wow. I added the soundbar/subwoofer to my shopping cart.
Midway through the fourth week of May, Crutchfield sent me an abandoned cart e-mail message, reminding me of the items I wanted to purchase.
In the first week of June, I added a mounting bracket to my cart, and I added a separate item not related to the television purchase to my cart. I placed my order, saving about $50 on the television, obtaining a soundbar/subwoofer valued at over $300 for free, and getting free shipping as per standard marketing practices at Crutchfield.
Ok, attribution experts. Tell me what caused this purchase?
- What impact did the mailed catalog have, given I already decided to buy the television prior to receiving the catalog, but given that the catalog was mailed a few weeks prior to buying the television (hint --- this is the crux of my stance on the "organic percentage").
- What impact did the first e-mail marketing message have, given that it featured the television I already placed in my shopping cart?
- What impact did the price change on the television have on my purchase?
- What impact did the offer of a free soundbar/subwoofer have on my purchase?
- What impact did the abandoned cart e-mail message have on my purchase?
- What impact did free shipping as a standard offer have on my purchase?
- What impact did Google have on my research?
- What impact did the fact that I already decided to buy this item before the marketing began have on my purchase?
- How would you, the marketing/attribution expert, parse this information without knowledge of the fact that I made up my mind on the purchase before any of the marketing activities were initiated?
Kevin: you are right on, and very funny in your account! (Thanks for a good laugh).
ReplyDeleteHere's the answer: you can't attribute, at least not for you. To me the biggest problem with attribution is that we cannot and do not know why a purchase decision was made because we have insufficient data. All we can do is measure what they did, and make *general* inferences about likelihoods that particular channels played a role. And we don't even know all the influences!
That stated, I still contend attribution is fine for reporting purposes (those who searched on google with keywords X, Y, Z followed by a banner ad clickthrough are 2x more likely to purchase). But it doesn't tell us why any *individual* did what they did.
But even though the situation is bad, frankly, while I'd rather know exactly why, I'm not that unhappy with the general trends if it results in attributing *more correctly* (not absolutely correctly). I'm just trying to make better decisions, and the improvement in understanding is still worth it, even with bad data and inaccurate inferences.
Nicely written, and (ditto Dean Abbott) very witty! And this is just for a B2C purchase, with one purchaser...consider the influence chains and dynamics for a B2B purchase, and the mind boggles.
ReplyDeleteI wonder what your thoughts are re: these issues for longer cycle B2B marketing efforts?
@LoriAusTex
To me this comes down to what it is I need to know to improve business performance.
ReplyDeleteSo to improve the profitability of the scenario you paint the key drivers would be reducing marketing costs, which seem to have been mini catalogue costs and working out if the bribe, sorry promotions, were excessive.
To achieve this I guess we need a good data set that captures the story you tell, and then plan some tests to see what happens if mailing is not sent or bribe is reduced.
So I need to know the story, but to improve performance I don't need to answer the question of who did what and should be credited with the score.
My first question is, how did you know Crutchfield offered free shipping (which seemed to cement your decision)- was it a website visit, or are you a returning customer who knows this is a standard policy? Ideally that could be quantified and included. My company went through an initial attribution study last year, and our next step is to try to understand where our marketing efforts attract and convert new customers vs. returning so we can further optimize our channels and campaigns (based on which type of customer we are trying to convert).
ReplyDeleteI would also categorize emails sent as a "trigger response" to items placed in a cart separately from emails sent to purely market to anyone, as those two activities are very different from one another.
Great post!
This is a fun read!
ReplyDeleteAttempting to speculate on any individual's cognitive process, even with post facto survey or interview data, is generally a fool's errand. Behavior tends to cement attitude - did you *really* fully commit to make the purchase on May 1st, or did this perception come about post-purchase? How would you account for this potential bias in survey/interview respondents?
Crutchfield has plenty of data here to assist with their attribution modeling:
Clickstream: Initial visit, subwoofer offer, purchase visit.
Emails: Email with television, abandoned cart message.
Catalog.
Transaction: this order, previous orders?
Every model is limited by the number and variation of data points collected and the amount of traffic to the conditions of the model. I don't suspect that their modeling is sophisticated enough to account for all of these "touches", but you can also bet that all of them are initiated because they have been demonstrated to convert (or you are part of an experiment that is attempting to unearth conditions for a higher conversion rate).
Good stuff as always, Kevin. And, though I make no claim to being an attribution expert, I think @MatthewTod hit it on the head. Is the attribution improving your business decisions? If you're suggesting that attribution modeling will never be perfect, I'm inclined to agree. However, if you're suggesting that attribution modeling has zero value in driving better business decisions...
ReplyDeleteIt's not a question of which element "caused" your purchase activity. It's more a question of whether any correlation exists between the various marketing activities and the desired business results. If you know that activity A correlates (in a statistical sense) positively with business result B, it's probably a good idea to engage in activity A, even if some other, unknown variable causes the result. We'd all love to know causation. But I think we'd all be willing to settle for "take action/get result" correlation.
Hi Kevin,
ReplyDeleteVery good example of why all attribution models are flawed. It is hard enough for anyone to be able to answer why they made a purchase let alone someone else defining this based on a subset of data (only some touchpoints).
It's nice to be able to allocate past revenue to different marketing campaigns, far more important to allocate your future marketing resources to maximise profitability. I think the key to this is not attribution but running experiments, testing different marketing mixes, defining value in research visits, using techniques to capture cross over between online and offline, etc.
By the way - why did it take over a month between purchase decision and actual purchase? Is there any chance that without one or more of the additional touchpoints, you may have ended up buying elsewhere (say if they had a big sale on)?
Cheers
Peter
--
Peter O'Neill
Founder& Lead Consultant
L3 Analytics
Mobile: +44 7843 617347
Website: www.l3analytics.com
Twitter: @peter_oneill
While I generally agree that correlation is something we settle for, it is also a dangerous thing to settle for. Correlations come and go, and we all have examples of spurious correlations that make for funny stories. We don't want spurious correlations because these are brittle (superbowl conference winner predicting S&P 500 up or down, for example). Causation is stable, but often unknowable.
ReplyDeleteWhat data mining tries to find often are correlations, yes, but more precisely, surrogates for the causal effect. In the case of attribution, while we don't know the cause, if we can identify other behavioral patterns that predict outcomes "reasonably well", that is, they get the gist of the cause but not precisely because they are surrogates after all.
How do we get there? We need enough data so that we overcome the noise and spurious correlations. When I build attribution models, this is key: avoid being fooled by spurious correlations with data. (This is where a good knowledge of statistics is very helpful--dealing with small data).
Kevin, I think knowing that you encountered multiple campaigns before finally placing the order is much better than foolishly thinking that the last cart-abandonment email did the magic of closing the sale.
ReplyDeleteHi Kevin,
ReplyDeleteVery good example of why all attribution models are to a degree flawed. It is hard enough for anyone to be able to answer why they made a purchase let alone someone else defining this based on a subset of data (only some touchpoints).
It's nice to be able to allocate past revenue to different marketing campaigns, far more important to allocate your future marketing resources to maximise profitability. I think the key to this is not attribution but running experiments, testing different marketing mixes,
defining value in research visits, using techniques to capture cross over between online and offline, etc.
By the way - why did it take over a month between purchase decision and actual purchase? Is there any chance that without one or more of the additional touchpoints, you may have ended up buying elsewhere (say if they had a big sale on)?
Cheers
Peter
All of the comments are interesting, that's for sure! Thank you to everybody for commenting.
ReplyDeleteHere's what I want for all of you to consider, because I think it is really important ... I was going to purchase this television from Crutchfield, no matter what.
I am building a home, so I need a television to hang on the wall. I simply delayed my purchase because I wanted the item shipped in a specific window so the builder could simply hang it for me as one of the final steps of the building process.
In other words, all of the marketing that happened post-decision was meaningless. Attributing any of it to my purchase over-states the importance of marketing. I made up my mind a long time ago ... as long as pricing was reasonably comparable to other brands, I was buying from Crutchfield because they offered free shipping.
In other words ...
(1) Attributing any of this order to the catalog I received is wrong.
(2) Attributing any of this order to search is probably wrong.
(3) Attributing any of this order to e-mail campaigns is wrong.
(4) Attributing any of this order to shopping cart abandonment e-mail campaigns is wrong.
(5) Attributing any of this order to the price reduction is wrong.
(6) Attributing any of this order to offering me a free $300 soundbar system is wrong.
This is the essence of what I talk about on this blog. If we are ever going to be good at attribution, we must first estimate what I call the "organic percentage" ... this is the percentage of orders that happen without marketing, orders that happen because of brand loyalty.
In my case, 100% of my demand is "organic". Once you know that, you attribute 0% to catalogs, to e-mail marketing, to shopping cart abandonment programs, to the soundbar promotion, to the price decrease.
That's what I want for you to learn from this exercise.
Does what I am saying make sense?
So the question is more: How do you determine that "organic percentage" with all that noise around? How do you discount it from all the other participants on this attribution contest?
ReplyDeleteWill introducing a post purchase survey where you ask a question that try to determine the cause behind the purchase help? What if we have:
1) One question where you present a list of all the possible marketing initiatives. Multiple choice.
2) A second question, later on, that asks bluntly if they made the purchase decision before receiving any marketing material or promotion?
If you have enough data, would this be significant enough to extrapolate the numbers and discount the conversions from the other channels and attribute it to "Orrganic" instead? What do you think Kevin?
Makes perfect sense. And even worse, if you are "organic", we have no idea if your purchase was due to a friend's conversation, a visit to a store, or driven by non-cookie-tracked searches.
ReplyDeleteThe best we can do, I argue here, is to measure organic vs. channel-driven purchases. Of course we will never know causation when a buyer (1) clicks through a banner ad he/she happens to see but would have gone directly to the site to purchase or (2) those users who never type the URL directly and only (imperfectly) google the company. But we do know organic conversion rates vs. channel-driven rates, and moreover we can track which pageviews were initiated by organic vs. various mediums.
Cructhfields marketing tactics are effecting their ability to monitor and track purchases which they are intended to make better.
ReplyDeleteIdeally, i believe you have to split the attribution across all channels, because each marketing message affected you. and ultimately you didnt but the TV when you placed it into the cart on first glance. Each of these marketing activites helped achieve the purchase.
My question is, if you were ultimately decided to but the TV from crutchfield on May 1..why did the purchase not occur until the end of may.
Hi Kyle, I did not purchase until the end of May because I wanted the delivery window for the TV to be congruent with when the builder could install the TV in my new home. The decision was made May 1, I simply waited for a month so that delivery was during the appropriate week.
ReplyDeleteJose --- if Crutchfield executes simple catalog and e-mail mail/holdout tests, coupled with varied paid search budgets by month, they can do a fantastic job of calculating the organic percentage, and be far ahead of the competition on measuring attribution.
ReplyDeleteKevin, I thought you were going to say you delayed your purchase to see how many marketing messages it would trigger, and to use this example in your blog post. :-)
ReplyDeleteCurious to get your thoughts on the impact of a deadline event (say, free shipping expires May 31) as well in terms of affecting attribution in a purchase lifecycle. I deal with this scenario and feel as though some of our marketing campaigns simply step in front of the customer in the final hours of a well publicized deadline- and look as though they drive conversions, when really the timing of the message coincides with the impending removal of the product from the marketplace.
Deadlines > Marketing Campaigns ... or they are part of marketing campaigns! Anyway, the deadline needs to be part of the attribution process.
ReplyDeleteAs you can see, this all becomes mostly pointless, because we can't get inside the mind of an individual to see what motivated the individual, without appropriate holdout tests in place (i.e. one set of customers is not given the deadline).
A purchase delayed is a purchase denied.
ReplyDeleteIf you didn't make a purchase, then you didn't have firm intentions and your whole contention falls apart.
Even if you had vague intentions of purchasing something from someone, you could have the same enthusiasm for Crutchfield and that specific product and postponed that purchase indefinitely.
- If something else had come up and you needed the money for something else--the brakes on your car needed to be replaced, a family member was hospitalized--then those vague intentions of purchase would have resulted in revenue for Crutchfield.
- If you'd used a comparison shopping engine and found the same model available from a reputable seller at a lower price with the same free shipping options, you'd probably have spent your money elsewhere. Again, Crutchfield would have ended up missing on the sale.
Until you finally submitted the order, you weren't entirely persuaded to make the purchase.
There's two ways to approach this--either the last communication closed the deal (last touch), or all the touches contributed to your purchase (the first catalog kept Crutchfield top of mind and drove your direct load site visit, email messages with reduced prices and kickers kept you interested and focused on a particular vendor).
Marketers who assume that 50% of their customers would have found a way to purchase regardless of the barriers to purchase or the competitive environment will find themselves losing market share and with dwindling resources because the aren't justifying their marketing spend. Knowing what messages have touched the customers who purchased and what touched the customers who didn't is the best way to understand how effectively you're influencing their behavior.
Brian J --- in the comments, twice, I mentioned that I delayed the purchase in order to allow delivery of the television to my new home within an acceptable delivery window. My purchase decision had nothing to do with your hypothesis about firm intentions.
ReplyDeleteThanks,
Kevin
A very astute practical example of one of the most poignant facts in analytics & statistics ... causation is not correlation.
ReplyDeleteAnswer is simple. Now that Crutchfield knows (based on this blog) that you had already made up your mind early on based on their free shipping marketing, they'll want to attribute a majority of your purchase credit towards their free shipping efforts.
ReplyDeleteThe information that you are giving Crutchfield through this blog is pure gold for an advertiser. By having a consumer explain their EXACT intent to purchase is what all advertisers want to hear!
Question now is, where did you hear about the free shipping offer? Through a friend? If so, than that doesn't really help Crutchfield. Through an ad? If so, Crutchfield can act on that.
In full agreement that most attribution models fail in this scenario. Also in full agreement that a post purchase survey, as long as filled out honestly by the recipient, will lead to what caused the purchase. All the attribution models I've seen do fail in this case: First touch, Last Touch, Rolling last 5 touch points, U-Shape Attribution (40% first, 20% middle, 40% last).
ReplyDeleteIdeally, Crutchfield has enough data in their attribution modeling that Kevin's case is normalized through data and the impact of his purchase does not allocate budget to the wrong marketing channel. Although, in this case Kevin provided a raise for virtually every department at Crutchfield.
What is not measured throughout this process is the value the customer put on free shipping and brand loyalty. Kevin had a perception and this perception hit his value points and he was going to purchase. However, each touch point didn't necessarily swing him one way or another. There is another aspect of this that is missing. The 3rd party vendors who's cookie data can lead to the answers Crutchfield cannot see in their website visitor data. This is DMP data, Panel Data, and Survey data. In today's market, you really have to track the purchase funnel through multiple views in order to get the best answers for your marketing spend.
Kevin,
ReplyDeleteI think you offer a great perspective but believe it to be only one of many. Here are the assumptions you are making:
All people shop like you - or that you are representative of the average person shopping for televisions (and therefore their marketing tactics are a waste).
During your "research" there wasn't a possibility that you could have been marketed to and converted by another retailer.
The Crutchfield marketing channels that you interacted with didn't affect your current or future purchasing decisions.
From Crutchfield's perspective, there could be a better way to attribute your sale.
We can't read people's minds. We can only make educated decisions based off from the data we have.
I realize that your point here is to question the validity of the data we have and how we are attributing sales to our marketing efforts.
But it is impossible for us to know exactly when that purchasing decision gets made. So until Adobe/Google/Coremetrics comes out with brain analytics tools, I'm happy using models to guess.
http://www.humbledmarketer.com
How come almost nobody in this comment stream answers the questions I asked in the post? It is perfectly reasonable to defend attribution, I mean, I get that you are attribution professionals so I realize you're going to defend your craft, that's reasonable.
ReplyDeleteBut folks don't answer the questions asked, and that is disappointing.
And almost nobody has chosen to address the concept of organic demand, demand that happens regardless of advertising, demand that can only be tracked by catalog/e-mail mail/holdout tests. How come nobody addresses this?
Of course it is impossible to understand what happens inside the brain of the customer, I'm not asking anybody to know that. Please move beyond that, and focus on the subtleties of the questions, and discuss how you'd address those subtleties. This is what attribution is all about, and so far, almost nobody wants to actually talk about what attribution is really all about!!!!
I didn't think those were actual questions - they seemed rhetorical in nature. Most of what you ask is still (as of June 2011) fairly debated. Even companies that provide full time attribution services are still using algorithmic models to determine % attribution to each channel touch (and these tend to range in balance between the traditional "last touch" approach and a predictive guess).
ReplyDeleteBut I'll do my best:
What impact did the mailed catalog have, given I already decided to buy the television prior to receiving the catalog, but given that the catalog was mailed a few weeks prior to buying the television (hint --- this is the crux of my stance on the "organic percentage").
I don't regularly read your blog and am unfamiliar with this "organic percentage." From context, it seems like you are referring to the % of total sales within a given time frame that are organic - that is, occur without being touched by a marketing channel.
From an attribution perspective, unless you entered a code from the catalog, when you hit Crutchfield's page, they would have tracked any revenue / visits from that session to an organic channel (assuming you directly typed the web address into the browser or clicked on a bookmark AND assuming that you hadn't recently visited the site via another marketing channel. "Recently" depends on how they define their cookie persistence window).
So unless you're able to tie your offline catalog data to online visitors, from an attribution perspective the effect of the catalog on your visit is nil.
What impact did the first e-mail marketing message have, given that it featured the television I already placed in my shopping cart?
Shopping Cart abandonment emails tend to be fairly effective. The answer to this depends on the model you're using. With last click, that email now owns 100% of your sale (which is obviously misleading and incorrect).
What impact did the price change on the television have on my purchase?
Well in your specific situation, I'm guessing this sealed the deal. Since you're shopping at Crutchfield because you receive free shipping, price seems to be a large factor in your purchasing decision. But quantifying the affect of this could only be estimated by running a controlled test.
What impact did the offer of a free soundbar/subwoofer have on my purchase?
See above (test).
What impact did the abandoned cart e-mail message have on my purchase?
Test it / model it.
What impact did free shipping as a standard offer have on my purchase?
So here's an interesting question. You knew about the free shipping offer before you even came to the site. So do you attribute this sale to a previous promotion you were running?
What impact did Google have on my research?
Cop-out answer: estimated using a model.
What impact did the fact that I already decided to buy this item before the marketing began have on my purchase?
It has a major impact but we can't quantify it.
How would you, the marketing/attribution expert, parse this information without knowledge of the fact that I made up my mind on the purchase before any of the marketing activities were initiated?
I'd probably use an attribution model to estimate the relative contribution of each touchpoint. Again, this is a model though, so it's an estimate. That's why testing is so important.
For a lot of quantitatively driven marketers, customer surveys are a valuable and overlooked resource. I think a lot of your questions could be answered by a robust survey program that could be used to inform tests.
Hi Matt, the organic percentage is the percentage of demand that happens without advertising. In the case of your business, you don't attribute e-mail sales to people who received e-mail campaigns ... instead, you execute mail/holdout e-mail tests, and then attribute the incremental revenue generated by e-mail campaigns.
ReplyDeleteMailed = $10.00 of store + e-commerce sales.
Not Mail = $9.50 of store + e-commerce sales.
Increment = $0.50 of store + e-commerce sales.
This is the trust of my message, and almost all of the attribution experts commenting on Twitter, sending me messages via e-mail, or leaving comments missed.
A business like Crutchfield (or your business) can use mail/holdout results for catalogs, for e-mail campaigns, and can vary monthly search budgets, and basically have the attribution nut solved by the test results. I am continually amazed that attribution experts won't execute the tests ... they'd rather make mathematical (and inaccurate) guesses than execute tests that give you a far more accurate answer.
My goal was to ask questions that were rhetorical, then I hoped that an attribution expert would mention that they would execute mail/holdout tests in catalogs, in e-mail, and would significantly vary paid search spend by month in an effort to obtain answers to the questions I asked.
I am thankful that you, Matt, gave a nice effort, much appreciated!
Not sure about the allocation of credit, but I can tell you what didn't drive your sale, the very last click. Which is where most marketers are in terms of making decisions. Forget how to allocate, most people just want the luxury of that problem.
ReplyDeleteThis is a very interesting stream. I am interested in knowing where you found out about the free shipping. It seems to me that this is not an "organic" purchase, but a purchase based on the free shipping promotion. To me the question of attribution belongs to the free shipping offer and the means by which you heard about it.
ReplyDeleteKevin: well stated as usual. Test test test.... I am always amazed at customers who don't want to have a control sample because "it will lose money". How do you know unless you test it? How do you know how much loss there is (i.e., what the attributable lift actually is) without testing for it? That's why I like the new uplift/netlift movement in the predictive analytics space--they are at least trying to get at these answers.
ReplyDeleteViolent agreement from Kevin and Dean about testing - that is the crux of understanding the impact for each event in the journey. I also agree that some organic factors need to be evaluated in any attribution model.
ReplyDeleteWhat I found intriguing was the additional promotion of the free $300 soundbar/sub-woofer...Kevin, your words - "wow" an unexpected, pleasant surprise. It certainly seemed to lock in your decision to move forward by adding this item to your cart (testing would be more concrete). I wonder also how this impacted your feelings toward Crutchfield as a brand for follow-on purchases or just recommending Crutchfield to friends in the future?
Impressed ... but made no difference in terms of the purchase.
ReplyDelete