Lies, Damned Lies and Mis-understood Statistics

Earlier this week, All Facebook featured a post on a report by DDB and OpinionWay examining the reasons that Facebook users “unlike” brand pages.

The key findings, as reported by All Facebook (the report appears to have been taken down from SlideShare, so I can’t link to it):

• The brand was no longer of interest to me (49 percent);
• The information available was not interesting (46 percent);
• Information was published too often (36 percent);
• The brand published information I did not appreciate (27 percent)
• Information was not published often enough (14 percent).

Interesting, useful data.

This follow-up chart in the post, however, is next to useless.

Why is this chart useless?

Because the sample size is too small for this kind of segmentation.

The original data is useful because the analysis is conducted at an aggregate level, over 630 respondents. At that sample size, we’re looking at a 3.9% margin of error at a 95% confidence level. That means, while there may be some variation among the top results, they’re useful at a high level.

Dig down to a country level like the chart above, though, and things start to fall apart. With a sample size of 78, given the number of Facebook users in the United States (155,746,780 according to Facebook), the margin of error for the US numbers is over 11% at that same confidence level. It’s not just the US, either – the margin of error for the France numbers is over 8.5%. Despite this, there’s no mention of these details on the post or comments; just an assumption that the numbers are correct.

This is a great example of why I think math is a critical skill for PR professionals.

PR pros need to understand the difference between valid statistics and invalid ones, so they can take advantage of useful information (like that at the top of this post) and disregard the non-useful stuff (like the regional breakdown above). What’s more, they need to know what’s news and what’s non-news too, so they can make an informed decision on what to pitch as the former and what to advise their clients to pass on promoting.

Are you comfortable reading between the lines when it comes to statistics? If not, it might be time to brush up.