Channels Aren't What They Appear

I introduced a brief quiz on Twitter regarding marketing channel performance. The answers I received were generally unsatisfactory.

Here's the gist of the question. A "brand" executed an on/off test. In Boise paid search is "on", in Salt Lake City paid search is turned "off" for a period of time. Matched market test.

For the month of the test, here's sales performance by major marketing channel.

Key facts from the test:

• When paid search was turned off, organic search results improved.
• When paid search was turned off, email / display / paid social / other website revenue marginally improved.
• When evaluating paid search by itself, it generated $10,000 in sales.
• When evaluating paid search in combination with other channels, paid search generated $2,000 in incremental sales.
• When evaluating paid search in combination with other channels, paid search expenditure resulted in a loss, not profit, of $4,300.

Questions for you:

• Which numbers from the table above would you share with your Executive Team?
• Do you believe the results of the test above? If the answer is "no", what does that say about you as a "data-driven" marketer?
• How would you adjust your marketing strategy based on the results of this test?
• What attribution rules would you implement as a result of this test?

Please discuss.