Measurement Technique

Back in 2007 I was invited to speak at a roundtable discussion at NEMOA. The host asked panelists a simple question.

• "How will you measure success in the future?"

When it was my turn to answer, I told the audience that holdout tests and spending experiments were the future. I also told the audience that what I just described happened long ago - in the "past" ... that leading companies conducted experiments and knew more than anybody else ... I described a year-long 2^7 factorial design used at Lands' End in 1993 (that's 25 years ago for crying out loud) to measure the incremental value of every business unit.

The audience, a room full of classic catalogers and the vendors who supported them, let out one of those audible groans that as a speaker you hope to achieve ... it means people are actively disagreeing with you and are actually thinking at the same time whether the speaker is right or the entire room is right.

Right after I said what I said, a vendor pointed out that Catalog Matchbacks were a "best practice". The room was soooooooo happy. Heads nodded. Professionals gleefully exchanged glances, knowing that forcing the catalog to "take credit" for the sales generated by an email marketing campaign was a best practice even if it was the wrong thing to do.

It's 2018.

How do you answer the question in the slide above? Do you answer with a "yes", gleefully holding your company to best practices from 2001, or do you conduct spending experiments and holdout tests (for all email campaigns and print campaigns)?

Your answer says a lot about how you prioritize channels at your company.