Social media platforms have become the go-to datasets for the modern organization wishing to understand popular perception about an issue. Why spend the time and money to conduct an opinion poll with proper survey design and sampling techniques when a quick keyword search on Twitter will yield a timeline and a word cloud that answers the question? Even better, that survey might only sample thousands of people, while that Twitter analysis might represent tens of millions of tweets. Unfortunately for the social media analysts of the world, things are not quite that simple.
Constructing and executing robust surveys and opinion polls is a complex, time-consuming and expensive undertaking. Each word carries enormous weight and even the ordering of questions with respect to one another can have a profound impact on the survey outcomes. Sample construction and reaching the necessary demographic, ideological and other membership density can require intricate methodologies and exhaustive workflows. Result weighing and evaluation requires extensive statistical skill.
A robust survey is not created by hurriedly typing up a few questions and releasing them via a Web form. It requires deep expertise not found at many companies.
For precisely this reason, social media analytics have emerged as the go-to replacement for the venerated opinion survey.
Most social media analytics platforms offer one-click survey execution - all that’s needed is the keyword. In contrast to the Excel spreadsheets, basic graphs and pages upon pages of statistical readouts that result from a professional survey, a Twitter analysis is typically presented in a slick beautiful dashboard that permits interactive drilldown and can be consumed directly by a senior executive.
The problem is that accessible public social media like Twitter still represent just a minute fraction of the earth’s population and those platforms are increasingly shrinking and shifting from content insights to behavioral insights. This means that far from representing society at large, they capture merely a small ever-shrinking sliver of it with unknown demographic biases.
Moreover, as survey designers learned long ago, social media is plagued by the problem that the most vocal users are typically those on the extremes of an issue rather than those in the middle. The majority of people that buy a product and are satisfied with it are unlikely to comment on Twitter about their purchase. In contrast, those who had a particularly bad or particularly good experience are more likely to share their thoughts.
A company searching Twitter for commentary about their products are likely to primarily encounter the small number of customers on either end of the spectrum rather than the vast majority of their customers in the middle. This can severely skew how they understand consumer preferences around their brand.
Similarly, when it comes to almost any social issue like politics, it is the highly charged and polarized activists that are most likely to saturate social media with their opinions, while ordinary citizens are less likely to live-stream their every political opinion to the world.
Self-censorship, once primarily discussed by those polling repressive regimes, has become a central issue in assessing contentious issues in our increasingly partisan and fractured societies. Questions like whether a person supports Donald Trump or embraces Brexit are all topics that can adversely affect a person’s employment, relationships and friendships, meaning users are less likely to answer truthfully in public settings like social media.
The anonymity of social media can also make it difficult to determine whether a user’s commentary is actually relevant or truthful. One of the reasons that some e-commerce Websites have incorporated “verified purchase” notations is to add additional authenticity to that user’s commentary as having actually bought the product they are reviewing. Similarly, when it comes to political polls, it can be difficult to assess whether a commentator is actually eligible to vote in the election they are commenting on, meaning their opinion may be entirely irrelevant to the voting outcome.
Professional opinion polls, on the other hand, incorporate a host of survey design methodologies to address these limitations, constructing samples that adequately represent eligible population demographics and go to great lengths to assess and correct the representativeness of their insights.
Putting this all together, social media has become the go-to medium for organizations looking to understand global reactions and opinions. The problem is that its ease-of-use and slick, simple outputs mask the myriad methodological limitations that severely bias its findings to the point of meaninglessness.
Much as our AI revolution has led to biased and brittle models due to an overreliance on free data, so too has the move from proper survey design to keyword social searches undermined the insights that drive our businesses and governments today.
In the end, not every question can be answered using a keyword search.
