November 01, 2010

E-Mail: This Isn't Hard

Here's what I'd like for you to try.

Step 1: Identify your e-mail marketing audience.

Step 2: Split this audience into thirds.

Step 3: In your November 8 campaign, send the first third your normal campaign message. Send the second third something totally different, unique, you know, something you wouldn't normally do. Just do it. Ask for forgiveness later. And then there is the final third ... DO NOT MAIL THEM ANYTHING.

Now, let the next week pass ... don't do anything ... DO NOT EVALUATE ANYTHING VIA OPEN RATES, CLICK-THROUGH RATES, AND CONVERSION RATES, because those metrics are absolutely fraudulent.

Once you have purchase data in your customer database through November 15, it is time to analyze your test. Your job is to only analyze ONE METRIC ... $ spent per recipient from 11/8 - 11/15.
  • Normal Campaign = $2.00.
  • "Different" Campaign = $2.00.
  • No Campaign = $1.70.
Let's say that these are the results.

Most important is the "Normal Campaign" vs. "No Campaign". You see a $0.30 difference. That's what matters, folks, that the "lift" you get when you send e-mail to a customer. This is the true impact of your normal e-mail campaign. You see, open rates, click-through rates, and conversion rates fail to account for a veritable plethora of customer behavior that lift actually measures. For a decade, e-mail marketers have measured everything wrong, and are now reaping the rewards of poor measurement techniques with ever-declining open/click/conversion rates. By measuring via "lift", you truly account for the impact of e-mail marketing.

And then take a look at your "different" campaign. If it performs as well as the campaigns you pour your heart and soul into, well, then, what does that actually say about what you actually pour your heart and soul into?

E-Mail Marketing ... this isn't hard. For once, abandon fraudulent metrics like opens/clicks/conversions, and instead focus on "lift", which is a true and accurate measure of e-mail marketing performance.


  1. Kevin,

    Love your stuff as usual and the points you make are right on. I do feel obligated to defend my kind (web guys) and say that 'conversion rate' used to - by definition - mean there was a sale and money in your pocket. I'd agree that today it is has become contaminated by stupidity and weird ideas about engagement and silly stuff like that there. I liked the world a bit better when conversion rate commonly meant a documented sale and nothing else.

    All the best,


  2. Contaminated is a good way to describe what happened!

  3. @Ross: Engagement is absolutely a valid goal for many non e-commerce websites.

    Only a tiny tiny percentage of visitors are prepared to buy from you, especially if you're offering expensive services. Sure, you can build your website to try to funnel people to a buy now form, but you're not going to get many responses.

    I think it's much better to provide content that your potential customers are interested in, engage them, prove your expertise, and make them happy. Then, when they're ready to buy what you provide, who do you think they'll seek out? Will it be the company with the hard sell approach, or the one that they've already had a positive experience with?

    That's why engagement is a valid goal in my opinion. Now, if you're selling products directly through ecommerce, I think an actual sale should be your primary conversion goal.

  4. E-mail marketers, the real question I'm asking is why don't folks execute the test illustrated in this post and measure via lift?

    That question seems to be dodged by folks.

  5. You are right-on with the comments, though in my experience working data mining/predictive analytics projects for marketing, no one dares to do it (no mailing to "good" prospects is considered a "loser").

    In the Predictive Analytics community, assessing data in this way is called "uplift modeling" (like here and here

  6. @Chris - I wouldn't disagree with your points about brochureware sites but to my mind the very definition of an Internet Entrepreneur is direct online sales; that's what makes it an Internet Retail operation.

    No doubt that great no strings attached content like newsletters is spectacular but the type of analysis suggested here by Kevin really is only applicable to Internet retailers. That being said, I know that my company site is surely brochureware, feel free to sign up for my no strings attached high quality newsletter Web Enlightenment. :-)

    You are preaching to the choir on best practices for services sites like yours and mine, but I think any conversation about lift calculation inherently includes a direct sale.

    On a separate but related topic I've been thinking about whether or not firms would respond to bribery – aka if I pay you this cash over here will you perform some real tests that include do not email, do not mail groups? I know, this is a wrongheaded idea and I'd never do it but it is tempting some days.

  7. I think we're all guilty of over complicating stuff. I suppose at the end of the day it's as simple as sales in and money made from each offer.

  8. This is ridiculous and offensive to your fellow marketers.

    Given what you've said here, I have to question your understanding of marketing in general and email marketing in particular.

  9. Hi Trey, I had a chance to review your comment and the blog post you wrote in response. Thank you for your feedback, good luck moving forward.

  10. Agree the measured sales lift is the most important metric but do not understand why open and click are "fraudulent" ... are you saying they are not as important or technically wrong ... if contaminated ... with what?

  11. I wholeheartedly agree marketers over-analyze something that should be as simple as "did your audience come back and buy from you?" Open rate / click-through / conversion is supposed to measure the effectiveness of different messaging approaches week to week, but instead it's used to measure the effectiveness of the entire email marketing branch - as in "how did emails do this week?"

    Here's a thought: lead generation marketers rarely deal with return visitors, so "lift" just means "how effective this week's campaign was at roping in new visitors versus last week's campaign." I'm curious to hear someone's take on that.

  12. I think that's reasonable for lead generation marketers --- though that's not my field of expertise, I'm an e-commerce, retail, and catalog person.

    In my world, measuring via mail/holdout groups yields more accurate results than open/click/convert. Some who have left comments differ, and that's fine, they may have different experiences than I have had that validate their point of view. I've done hundreds of mail/holdout tests, the results in my field are more interesting, and lead to a better measure of profit, especially when the holdouts are done over a month or a quarter.


Note: Only a member of this blog may post a comment.

I See Dead People

From LinkedIn, where I wrote this on Sunday: