Who's Nielsen Trying To Fool?

Like most of you, I would like some reliable data about social media that I can take to my clients and say, "Look, social media isn't all hype. Here's some data to show that it really works."

Yesterday I finally thought I had some.

I received an email from Nielsen informing me that they were "releasing the results of an important new study."

The report is called "Understanding the Value of a Social Media Impression" and purports to quantify for us the value of "paid" and "earned" media advertising on Facebook.

Unfortunately, from what I can tell, the report is complete bullshit.

You have to remember that I'm not a researcher and I hope some of my readers who are researchers will look at this study and tell me if my qualms about it are correct or not.

You can find the study by going here and following the instructions.

The first thing that caught my eye was this sentence in the email:

"Through our joint partnership with Facebook we are committed to helping marketers fully leverage social media to build their brands."

Right away a red flag went up. Honest researchers aren't "committed" to anything other than reporting the truth. Researchers with agendas and commitments aren't doing research, they're doing advocacy.

It just kept getting worse. In the report itself, they state...

"We have now conducted six months of research consisting of... more than 125 individual Facebook advertising campaigns from 70 brand advertisers..."

The implication is that this study is the result of all that research. Otherwise, why mention it?

It's not until four pages later that we learn...

"For the purposes of this case study, we examined the impact of Facebook advertising on 14 campaigns...Campaigns selected were representative of more successful campaigns ..."

WHAT?

In other words, of the "more than 125 individual Facebook advertising campaigns" they have studied, they only picked 14 of the top performers for this "case study" (and notice how it has mutated from an "important new study" to a "case study" so they have a loophole to crawl through when this baloney hits the fan.)

So, let's recap. You take only the top 11% of the campaigns, you ignore all the failures and all the mediocrities, you analyze only these top performers and you present the results as if they prove something?

"This study demonstrates that advertising in the social context works for brands and works well."

This isn't research. This is a new business pitch.

Update:
For a more mathematically oriented interpretation of the Nielsen/Facebook study, I recommend this from Ben Kunz at Thought Gadgets.