Social media has a terrorism problem. From Twitter’s infamous 2015 proclamation that it would never censor a terrorist to Facebook’s long delay in adopting signature-based content blacklisting, social media has become a critical inadvertent ally in helping terror organizations throughout the globe recruit, communicate and promote. The platforms themselves have been slow to respond, initially rejecting calls to remove terrorists from their walled gardens, before reversing and aggressively embracing the idea of purging violent users. However, for all their public discourse, the platforms have taken little concrete action, reflecting both the economic realities that they have little incentive to invest in content moderation and the real-world complication that deleting terrorist content requires understanding context, not blindly deleting any post with a given keyword.
Over the past few years the major social platforms have rapidly evolved from defending free speech at all costs towards recognizing that the public and policymakers do not take kindly to their systems being used to help encourage, support and direct violence. Towards that end, most platforms have centralized on a two-pronged approach to countering terroristic use of their tools: human review and automated blacklisting of content that has been previously deleted by a human reviewer. Platforms are also experimenting with a third category of automated scanning that can, in theory, flag novel posts that are likely encouraging or promoting terrorism and route those for human review, but for the moment those are largely limited to a few platforms and focused nearly exclusively on textual content.
Human review is the gold standard of content moderation, but has many drawbacks, chief among them the very limited scalability of using even tens of thousands of humans to attempt to moderate billions of daily posts. The difficulty of staffing sufficiently large teams of moderators that represent all of the languages and cultures using social platforms makes comprehensive human moderation an all but impossible task. As Facebook discovered in Burma, a handful of moderators cannot begin to cope with the firehose of content from any given location on earth. To effectively understand the context and symbolism of the content they are reviewing, moderators must also have strong cultural roots in the geographic areas they are assigned. Assigning an American who has never left the country and learned basic comprehensibility of a language in school to be a moderator for that country’s posts has little chance of success.
There is also a very real psychological cost in using humans to review horrific content.
Historically, most companies relied on a post-review takedown approach, meaning that if 1,000 people all upload the same terrorist propaganda image, all 1,000 copies of the image would have to be individually reviewed by different moderators and deleted one by one. The same user could then simply reupload the image seconds later and it would have to be flagged and reviewed once again. When asked previously why Facebook did not use signature-based blacklisting to prevent reuploads of content deleted by moderators, the company would simply decline to comment.
Growing public and governmental pressure finally forced the major social platforms to take concrete action, leading to a shared initiative in which the companies agreed to jointly implement signature-based blacklisting and share a joint list of image signatures. Under this model, when a moderator deletes an image, a unique signature or “digital hash” of the image is entered into a central database and used to prevent the image from being reuploaded in future on both that social platform and any others that are a member of the consortium.
Signature-based content removal has now become the defacto standard in Silicon Valley for removing terrorist content.
On paper, this kind of blind content blacklisting offers a computationally cheap and trivial solution that removes a considerable cross-section of terroristic content from circulation. Most importantly, it allows tech companies to argue that they are taking real steps towards curbing terroristic use of their systems.
Unfortunately, blind signature-based content removal is more public relations ploy than effective counter-terrorism tool and in fact can do more harm than good in restricting public discourse about terrorism, from news media coverage of attacks to victims documenting the atrocities they endured to civil society groups launching CVE initiatives.
Tech companies can rightfully claim they are preventing millions of reuploads of previously identified terrorist content, while sidestepping the far more important question of how to identify all of the new content being generated each day. For its part, Facebook now claims that 99.5% of the terrorist content it removes is through this signature-based removal and its other automated filters, reflecting that the majority of its success has come from removing preexisting known content, rather than new content being created every day.
The signature database underlying these efforts is almost exclusively focused on ISIS and Al Qaeda content and consists of less than a hundred thousand pieces of content, a portion of it duplicate copies that have been slightly modified.
Most importantly, signature-based blacklists fail to take into account the context in which a given post is shared. Imagine a new ISIS image that depicts the use of GPS-controlled drones to drop modified grenades without human intervention. The image would likely be initially shared by ISIS sympathizer accounts lauding the weapons’ lethality and thus flagged by human content moderators and added to the signature database. Subsequently, news outlets might use the image to discuss ISIS technological evolution and ways they could be subverted. Victims of the drone attacks might use the images to illustrate what happened to them. Civil society groups might use the images to condemn ISIS’ barbarity. Yet, all of these secondary uses would likely be blocked by blind signature blacklists that simply remove every subsequent use of the image, regardless of purpose.
Context is everything when it comes to the meaning of a given piece of content. Signature-based approaches cannot distinguish between a post lauding a terror attack and one condemning it. Yet, Facebook has previously clarified that it is nearly exclusively relying on signature-based removal for the imagery, video and audio content that forms much of the propaganda output of today’s social savvy terrorist organizations.
While Facebook has offered that it is using machine learning approaches to attempt to flag novel textual content, it has declined to comment on the accuracy of those tools, especially their false positive rates and has declined to commit to allowing external review of their accuracy.
This has created a landscape in which much of the counter-terrorism work of the major social media platforms is based on simply preventing users from reuploading a small database of previously identified image and video content, along with machine learning experiments of unknown accuracy and efficacy.
Signature based removal traces its roots to the automated systems designed originally to flag unauthorized reproduction of copyrighted content and the removal of child pornography. In the latter, there are no legal contexts under which such content can be shared for any purpose, including condemnation, making it easy to use context-free signature removal.
Things are far more complex when it comes to subjects like terrorism, where it is not the content itself that is illegal or objectional, but rather the context in which it is used.
Instead of blindly blocking all uploads of an image, platforms must look at the context of each of those posts. From my own experience processing half a billion images through Google’s Cloud Vision API and testing its Cloud Speech API’s ability to generate useable audio transcripts of ISIS videos with multiple users screaming in Arabic over gunfire and explosions, we have the tools today to sufficiently annotate novel imagery and video content to flag material that contains depictions of violence and to generate machine readable categorizations and transcripts that can be used to identify terrorism-related content.
Tools like Google’s Vision, Speech and Video APIs and customized AutoML models can take an image and recognize the presence of terrorist organization logos, OCR text in 50+ languages to render any subtitles, textual overlays and background text searchable, identify the specific make and model of weapons in the scene, recognize specific uniforms and insignia affiliated with terror groups, convert audio narratives into searchable transcripts and even identify the presence of violence, catching a small group of blood droplets in the corner or distinguishing a gun sitting on a table from one being pointed at a person.
Such tools are completely automated and extremely efficient, able to process an image in a fraction of a second and scale to billions of pieces of content a day.
Moreover, Google’s Vision API is able to perform a reverse Google Images search on each image, identifying all of the locations it has seen the image on the open web in the past, along with the captions used in each of those cases. This means that even if an innocuous image is uploaded that does not on the surface appear to depict anything terrorism-related, Vision API can flag that when the image has appeared elsewhere on the web it has always been described using captions that mention ISIS and terrorism. Thus, an ISIS recruiting video that features smiling people, but which never actually mentions the group by name can be readily identified entirely automatically.
Similarly, Google’s Vision API is able to recognize similar images, including images from which sections have been cropped to generate the image in question. Such tools can be readily expanded using image presegmentation to identify novel image composites built from cropped sections of previous images. Such composites will not typically be recognized by today’s signature tools, even if each source image is in the database.
Recognizing that an image contains an ISIS flag or that a video mentions joining ISIS in the narrative allows tools to move far beyond current signature-based approaches towards recognizing novel content, but still does not solve the context problem.
White listing recognized news outlets could help with the problem of an evening news broadcast including a clip of an ISIS video or a news article that includes an ISIS image as an illustration but does not address the broader problem of how to separate posts that laud violence versus condemning it.
One promising approach is to combine statistical sentiment assessments with neural categorization models to yield a hybrid score for each piece of content that assesses both its focus on the violence or recruitment elements of terrorism and whether it uses a voice that appears to promote rather than condemn or clinically document. Again, the ultimate arbitrator should always be a human with a deep cultural background and understanding, but such tools could help flag large volumes of novel content that today remains beyond the reach of current approaches.
Of course, even human reviewers do not solve the problem of deciding what constitutes “terroristic” speech. While most countries recognize ISIS and Al Qaeda as terrorist organizations, looking globally the problem is far more complex. Many independence groups have contested classifications, recognized by some nations as freedom fighters that are actively supported with funding, weapons and training, while targeted by others as violent terrorist organizations subject to sanctions and covert operations against their leaders. From personal experience running human classification projects in the past, getting a room full of subject experts to agree on whether a given post represents “terrorism” or a “human rights abuse” or “torture” can be nearly impossible, much as it might seem to the public and policymakers that such labels should be fairly straightforward to apply. Today it is the US that decides which groups are terrorist groups and thus should be removed, but will other countries demand a say in the future?
For example, should all posts relating to Hamas be deleted globally from all social platforms due to its classification as a terror organization by some countries?
Repressive regimes throughout the world are also increasingly exploring the use of counter terrorism legislation to classify government criticism as terroristic speech that they could then use legal mechanisms to compel the major social platforms to remove, much as DCMA has been misused in the past as a tool for stifling criticism.
Unfortunately, one of the reasons that Silicon Valley has so quickly latched onto signature-based blacklisting is that it is ill-equipped to take more serious steps towards more robust content removal. The data sciences groups at most companies tend to draw heavily from a handful of countries, operate primarily in English and have little experience with the languages and cultures they are tasked with examining. I’ve seen counter-terrorism data science groups at major social platforms that were comprised nearly exclusively of English speaking Americans with strong statistical and programming backgrounds, but nary a single field-experienced counterterrorism expert nor even a single person who actually can read a word of Arabic. Validating models often consists of testing how well they fit against existing training datasets, with machine translation and limited outside SME experts brought in for spot checks. Even those groups lucky enough to have Arabic speakers on staff tend to draw from non-native speakers who learned Arabic in school, rather than recognizing that the Arabic speaking world is an incredibly diverse and culturally rich place. Just as one would not grab a random American and say they can speak for the views of the entire US, Silicon Valley must recognize that the web does not consist of “English speakers” and “the rest of the world.”
Often, when meeting the data scientists assigned to build counter-terrorism tools, the first question one asks is how on earth their company expects such a group to meaningfully contribute to removing terroristic speech given their utter lack of qualifications. There is also little interaction at the data sciences level with the governmental field personnel who are on the front lines of identifying and analyzing terroristic speech and which can provide the most valuable insights into symbology and narratives and the trends they are observing. Cultural immersion is absolutely critical to understanding whether an innocent-seeming post actually says far more than it seems at first glance. Much as understanding Iranian political discourse years ago required not only an understanding of Farsi and advanced knowledge of Iranian poetry, but also a deep cultural immersion and understanding of the deeper meaning of each line, so too does effectively identifying and removing terroristic speech require more than an American who learned a few words of Arabic in school and has never left the US.
Looking globally, ISIS and Al Qaeda are far from the most impactful terror groups in many parts of the world. Combatting terrorism speech globally requires working across nearly every language in use on the social platforms and deep cultural expertise spanning the globe. It requires studying all recognized terror groups and building libraries of their symbology and narratives across all of those languages.
Instead of relying on a small central ISIS and Al Qaeda-focused database of images curated by the major social platforms, what if we created a new globally cooperative database of content contributed by governments, NGOs, civil society groups, academic and independent researchers and even ordinary civilians, documenting the visual, audio (such as music) and textual narratives used by each group and evolving patterns in their material? Such a database would pose unique oversight and curation concerns, but even if initially limited to major NGOs, researchers and civil society groups, could go a long way towards internationalizing terrorist content removal across terror groups, languages and narratives. It could also help expand filtering beyond public posts towards the vast archives of content the groups share through alternative channels.
Much of the focus of current efforts has been on the removal of content from social media platforms. The question of encrypted communication channels has largely been relegated to the encryption debate and whether companies should be forced to provide backdoors to their products for law enforcement and intelligence use.
Yet, the growing ability of even low power mobile devices to run complex deep learning models entirely on-device raises the possibility of blacklisting terrorist speech even in encrypted communications, by blocking it from being shared in the first place. One could imagine common encrypted communications tools building in basic filtering models that examine every message or file and prevent those relating to terrorism from being shared at all. Of course, bad actors will always find a way around such safeguards, but the more difficult you make it for terrorists or criminals to use such platforms, the more risk you introduce to their communications and the more they have to codeswitch between platforms and codes, making it more difficult to effectively conduct operational planning and recruitment over secure channels.
Of course, self-censoring communications channels are a repressive government’s dream and asking major chat apps to build in models that scan all communications and block those deemed unacceptable would almost immediately bring calls from governments to extend the models to block calls to action for legal protesting or criticism of government.
Similarly, as we get better about removing terrorist content and expand between ISIS and Al Qaeda, critical questions are raised about who decides which groups to silence and which to allow to speak?
An even bigger question is whether we should be silencing terrorists at all. Allowing them to communicate in the open, while targeting their more secure channels yields a greater exposure surface through which to observe their activities, influencers and narratives. Pushing this activity underground makes it more difficult to track. Moreover, as China has taught us, it is often far more effective to encourage self-censorship by drowning out speech than by playing whack-a-mole deleting it. Perhaps most effective of all is to take a page from the world of counterintelligence, creating a “wilderness of mirrors” in which terrorists and their followers and would-be recruits are no longer able to know who or what to trust, disrupting their information environment at a far deeper level that has lasting impacts on the ability of the organization to harness the digital world.
In the end, there is little incentive for Silicon Valley to invest in removing terroristic speech. Such content compromises only a microscopically small fraction of the total volume of billions of social posts per day, while the economic costs of hiring enough human reviewers and building enough machine learning models to filter those billions of posts to find the small number of terrorist posts is staggering. Short of government regulation there is little reason for them to do more than public relations ploys like signature-based filtering. Fundamentally rethinking social media CVE, expanding beyond ISIS and Al Qaeda, looking across all the world’s languages and cultures and, most importantly, evaluating the context of each post, is simply not cost effective for the companies today to invest the necessary resources in.
After all, for companies that can recognize a single person’s face out of billions in an instant, record every link we click and every post we look at, run ad auctions involving millions of variables billions of times a day and create digital dossiers that know us better than we know ourselves, it sure seems they could do a whole lot more than merely block reposts of a few tens of thousands of previously identified images and videos.
Putting this all together, we have the tools today to do far more in countering terroristic use of social media and the broader web, especially harnessing deep learning approaches to identify novel content and its context both in a given post and across the web itself. The question is when will Silicon Valley finally decide to bring its immense capabilities to bear on its terrorism problem.
