November 12, 2014

Twitter, ROI, And Bullshit

I’m a simple man. Annoying, but simple.

Among my simple needs is a yearning to understand things. Understanding something, science has taught us, turns out to be a lot different from thinking we understand it.

Much of what passes for “understanding” in the world of advertising and marketing is merely the correct memorization and repetition of phrases.

But knowing the words doesn’t mean you understand the concept. As Richard Feynman brilliantly pointed out, someone who knows the name of every plant knows nothing about plants. He knows what people call plants. Knowing the names of things is not the same as knowing things.

This struck me a few weeks ago when I was reading an (unbelievably naïve and amateurish) article in Ad Age titled Study: Twitter Marketing Drives $716 Million In Car Sales. It reported that Twitter recently released a study which showed that advertising on Twitter had accounted for over 700 million dollars in car sales. According to the story, for every dollar a luxury compact car brand invested in Twitter, they received a return of over 17 dollars.

My initial reaction to this was that it was unmitigated bullshit.

Without knowing anything about the actual calculation of the ROI in question, I am certain it is a case of mistaking correlation with causality (you can read about that here.) It reinforced my belief that almost every ROI number I have ever seen or read is, likewise, bullshit (you can read about that here.).

Here’s my simple-guy logic:

If you were to discover an investment that yielded you a return of 17 times your capital, why would you not invest every dollar you have in it? You’d have to be crazy not to.

If you could get a 17 to 1 return on dollars by putting them into Twitter, why not put every cent you have into Twitter?

I did some math. Let's say that during a 2 month period I invested $1,000 in marketing on Twitter. After two months I would have a return of $17,000.

Then let's say I re-invested it for another 2 months. And let's say I did this six times.

At the end of my little experiment I would have a return of $1,419,857,000 on my original $1,000 investment. A billion-and-a half on a thousand dollar investment. Man, this Twitter thing is awesome!

Intuitively, everyone knows these ROI numbers are bullshit. Which is why no one takes all their money and places it to win on so-called "return on investment."

According to Ad Age, the numbers in this case were calculated by a "marketing analytics company" that just happens to have a partnership with Twitter. If I didn't know about the enormous strength of character in the marketing industry, I might think that sounds a little cozy.

Here are some thoughts about ROI calculations:

1. Never believe the ROI calculation of an interested party.

2. Never believe the ROI calculation of a short-term promotion. It never accounts for all the zillions of dollars spent building the brand that made the promotion even possible. It may be a valid calculation of something, but it is not a valid ROI.

3. Unless you have a clear definition of the factors included in the ROI calculation, and the formula used to derive the calculation, you know nothing about ROI. You know something about what people call ROI.


Chris said...

Agree with what you're saying here Bob, but one thing strikes me - wouldn't we expect ROI to diminish as marketing investment increased? Surely there are only a finite number of people to reach?

Adam said...

My thoughts exactly, Chris. Diminishing returns. You might see good results when people see your TV ad 5 times. Making them see it 25 times is 5 times more expensive but the results most definitely won't be 5 times better.
This is not to defend the "study" in question. Working in direct response makes calculations much more straight forward. And the job much more difficult.

Dan Plant said...

"Knowing the names of things is not the same as knowing things"

That is my new favourite quote - currently scrawled on a post-it note on my monitor.

Neil Charles said...

I make a living measuring the ROI of marketing campaigns, so this and the linked 'Return On Ignorance' post make for somewhat uncomfortable reading.

They're not wrong, which is why they make for uncomfortable reading.

We need to be honest about what ROI measurement, as it exists today, does. Putting any awareness increases etc. to one side for a minute, all ROI metrics do the same thing: They measure the spike in sales that occurs during and in the few months following a marketing campaign.

1. This isn't the whole ROI from marketing.

2. It is still incredibly useful information, because you can find out which campaigns create bigger spikes.

You can also use this ROI for budget setting. In simple terms, when a CMO says "I want to make sales double this year by spending more on the same adverts", it allows an analyst to say "that won't work. Let's talk alternative strategies". ROI measurement as it exists today lets you build realistic, accurate, short to medium-term sales forecasts.

That said, a 17x ROI from social media is clearly nonsense for the reasons outlined in the post.

The marketing industry has a habit of responding to a common-sense analysis that it doesn't like, by making it more complicated. It doesn't invalidate the common-sense answer but everyone gets confused and then bored and then the problem goes away. Bringing diminishing returns into this one - as a couple of comments have - feels a lot like that to me.

Tom said...

The really uncomfortable thing about calculating ROI on advertising is that anyone not from Advertising can see how totally impossible it is, there are simply far far far too many variables.

For one, any new product can't possibly ever have an ROI attached to it, to do so we'd need to somehow know what would have happened without that ROI. It seems that Tesla were able to sell every car they ever made with no advertising. If Tesla had done one promoted Tweet to support their car, the ROI would seem to be infinite. Yet common sense makes us know this to be bull.

Secondly even existing products in markets where little changes still have far too many variables, it could be the weather, it could be competitor activity, regulation, new distribution. Anyone know knows anything about Marketing can see that it's impossible.

We need to rely on gut more. The worst thing in advertising was the illusion that we could measure it. You may like this.

Neil Charles said...

"For one, any new product can't possibly ever have an ROI attached to it, to do so we'd need to somehow know what would have happened without that ROI"

Spot on. Calculating ROI in the first 6-12 months of a product's life is pretty well impossible. Everything's going upwards and you can pin those increases on anything you like.

"Secondly even existing products in markets where little changes still have far too many variables"

That's not correct though. There's a very well established statistical technique called multiple regression, which can pick apart which factors are having what impact on sales. There are textbooks full of caveats about where it doesn't work and it needs huge care, but a decent analyst with high quality data can separate advertising's immediate sales impact from the weather, distribution changes and other factors, fairly easily.

The list of things that make a significant difference to what a brand sells in a week isn't actually that long - only 7 to 8 factors. As long as they've been varying over time you can work out which bits are having an effect.

Graham Strong said...

From the referenced AdAged article:

"With over 65,000 daily tweets about purchasing or researching a new
car, Twitter gives auto brands the unique opportunity to connect with an
audience of in-market shoppers who have expressed intent to buy a car,"
said Rob Pietsch, head of Twitter's auto vertical, in a statement. (Emphasis mine.) other words, they were already in the market for a car, so Twitter didn't "drive" sales, only (perhaps) redistributed them.

Swopard said...

ROI diminishes bur profit rises. If the campiagn is really effective you should still invest in it much more

Tom said...

I agree that by isolating some variables, using banks of empirical data we can get to a point where we know A works better than B or that when we did C, D tended to go up, but I don't think for one second it's as advanced as people assume. We do live in a world where the weather is more likely to affect how many people show up at your funeral than any other metric, where film reviews matter more than advertising films, where sometimes stuff just happens.

We'd be better to assume we know nothing and that we can get a vague sense of correlation to some activities that we do and the be pleasantly surprised by what we establish, that what we do now, which is to peddle the myth we can somehow measure ROI.

To buy a car probably includes 20,000 data points, of which 90% we don't even realize we're making, it's smell, how greasy the salespersons hands were, how nice the coffee was, to attribute a sale to ANY one thing is madness.

Neil Charles said...

"We do live in a world where the weather is more likely to affect how many people show up at your funeral than any other metric"

It it really? Or are you using a vague sense of correlation to reach that conclusion?

Tens of thousands of people turned up to Nelson Mandela's funeral and it was horrible weather. The rain certainly stopped those numbers being even higher, but most important factor in how many people turn up? No way.

Measurement is useful. Just don't try to push it beyond what it can actually do.

I can't prove it (intuition is useful too!) but bet a car salesman would disagree with your 20,000 data points and say it's almost always the same very few things that decide a sale.

Jeff Spicoli said...

"Any fool can know. The point is to understand." ~ Albert Einstein


How can you say that this tweet alone didn't sell at least $17k in Acuras: "Here's something you haven't seen before. Configure your dream 2015 Acura TLX below without ever leaving Twitter."

Jonathan Rodgers said...

Ha! Love it. I saw this crap the day it came out, and posted this comment:

Put it in perspective please, AdAge.

U.S. auto sales in 2013 were 564 billion dollars. Take 34% of that total (34% of auto sales was the number covered in this "study") and that gives you sales of 191 billion - upon which was studied Twitter's effect.

Twitter's 716 million dollar impact on 191 billion in sales amounted to .4 percent: less than half of one percent. Free doughnuts at the dealership probably have a bigger effect.

And the article advises: "for best results, mix with other marketing messages and media." Really? Ya think?

When will the social media hype and bullshit stop?

Kurt Jetta said...

Terrific article, as usual. Disagree with your Point #2. In fact short-term promotional ROI is about the only ROI that CAN be accurately measured. All of the "zillions of dollars spent building the brand" are already absorbed into the baseline estimate.

Kurt Jetta said...

What can definitely be confirmed is that Twitter drove an incremental purchase by me of your book after I saw the retweet of this article from Dr. Byron. 10 more people like me and you have a positive ROI on the time it took you to post the article!

Tom said...

Yes, I think bad weather puts people off attending funerals- causation not correllation.

You can be sure said salesperson doesn't attribute the sale to Twitter and 19,800 of those data points will have led that person to be in that showroom at that time, having that conversation.

Swopard said...

That's not so important, because if it's still positive - that means it generates net profit. Otherwise you are just saying: "I dont want any more profit, because my ROI will drop". the higher ROI the farther away from saturation you are - and farther away from potential, bigger net profit.

LeShann said...

Yes, but the profits will be impacted by the diminishing returns at some point.

SpectrumInsight said...

First, do people really take ROI claims such as this entirely at face value, any more than they believe fuel consumption figures from a car manufacturer? We reckon you'd be pretty stupid if you did but equally stupid not to look in more detail at what lies behind the claim in case it does have relevance and is appropriate to you. After all, a car's official fuel consumption figures will be bullshit but that doesn't mean they're completely useless in the decision-making process.

Second, it's obvious why you shouldn't invest every dollar in Twitter in this example regardless of the ROI, even if this were 170 to 1 let alone 17. Why? Because of the reasons stated in point 2.

No, the REAL problem with ROI isn't doing the math (or maths as we'd say in the UK). It's that ROI calculations, typically, are not simple. The problem then isn't that ROI calculations are bullshit but, rather, that people who need to understand them can't; and it's nearly always in the resulting simplification process that the bullshit appears.

None of this makes an ROI claim invalid. What it does do is demand that somebody looking at it asks the right questions to test its validity. Unfortunately, virtually no advertising professionals have a clue what those questions should be and that is just as much an indictment of the advertising profession as it is of the people making the ROI claim in the first place.