Perhaps the most Orwellian of all of Facebook’s presentations at F8 last week was a little-noticed presentation about the company’s huge investments towards performing content moderation directly on users’ phones, allowing Facebook to scan even encrypted person-to-person WhatsApp messages for content Facebook or repressive governments dislike. Today Facebook scans posts after they are uploaded to its central servers, but as the company moves towards a decentralized person-to-person encrypted communications world, it is aggressively eyeing moving content filtering directly to users’ phones. Coupled with previous statements from the company that it considers a user’s private non-uploaded camera, microphone and photo gallery to be fair game for Facebook to search without notification or permission, Facebook’s shift to the edge sounds so Orwellian it makes the NSA spying efforts disclosed by Edward Snowden sound like child’s play. Are we really rushing towards a world in which Facebook-produced AI algorithms will be running directly on our phones, monitoring every second of our lives, eliminating encryption, preventing us from seeing or saying anything the company doesn’t like and even sending alerts when we utter an unauthorized phrase? Is this truly to be our future?
At first glance, the F8 talk “Applying AI to Keep the Platform Safe” seems like a typical technical deep learning presentation. A series of Facebook engineers take the stage to talk about the complexities of running large advanced AI models on mobile hardware, from model compression and quantization to power consumption to minimizing content hashing signature databases to all of the usual topics found in the kind of engineering talks found at any AI conference.
Look a bit closer at the examples the company offers and the ramifications of the vision it presents and a whole new Orwellian world emerges.
As Facebook has publicly pivoted towards a new “privacy-first” vision of its platform, transitioning from public web host to private encrypted communications provider, there has been considerable discussion regarding how the company will be able to enforce its content moderation rules and create the behavioral and interest models it needs to sell ads, when everything that reaches its servers is encrypted.
I have long suggested that Facebook would inevitably turn to the edge to circumvent encryption, running its moderation and advertising models directly on users’ devices.
It turns out that is exactly what they intend to do.
Terrorist organizations have been rapid adopters of off-the-shelf encrypted messaging platforms, leveraging the protections of free military-grade encryption to shield their recruitment and operational planning from authorities.
This has presented a unique conundrum to social media companies whose products have been repurposed to support terrorism: how can they filter for terrorist content when everything that reaches their servers is encrypted?
The answer, as I’ve long noted, is to perform that filtering on users’ devices themselves. The original unencrypted message can be scanned on the sender’s phone before encrypting it for transmission and the decrypted message can be rescanned on the recipient’s device after it has been decrypted for display.
In its presentation, Facebook discusses its advances in performing content signature scanning directly on users’ phones. A database of signatures of known disallowed content is uploaded to the user’s phone and every message they attempt to send is scanned against this database directly on the user’s phone before the message is accepted for delivery to Facebook. Even encrypted messages can be scanned in this way prior to encryption, eliminating the ability of bad actors to share known illicit content through encrypted channels.
While Facebook uses the example of child exploitation content in its presentation, the same process could easily be used to combat the sharing of known terrorism content.
Yet, the company goes far beyond simple known content blocking, towards describing a future where its entire AI-powered content moderation infrastructure lives on users’ phones instead of Facebook’s servers. The company would regularly push out updates to a gallery of AI algorithms and signature databases that would reside on users’ phones and scan all of their content before it is posted or sent as a message.
A private text-only WhatsApp message between friends that violates Facebook’s content rules would be deleted on the sender’s device before it can ever be sent.
In short, by moving content moderation to the edge, Facebook will no longer be deleting bad content hours, days or even weeks after it has gone viral and spread to the far corners of the web – it will delete posts before they can ever be sent in the first place.
While few would shed tears about a terrorist no longer able to share recruitment propaganda or a hate group inciting violence against a minority group using WhatsApp, the fact that Facebook’s moderation algorithms are being built in the dark means we have no visibility into just what will constitute prohibited speech in the eyes of Facebook’s increasingly AI-driven future.
Will any discussion of government regulation of Facebook or new data privacy laws be banned as “unacceptable speech” and every encrypted WhatsApp message mentioning something negative about Facebook deleted before it can be sent?
It is unfortunately not a far stretch to see Facebook take its “logo use rules” that were at the root its deletion of Sen. Elizabeth Warren’s ads and encode those into an algorithm that deletes any private encrypted message that mentions the company or references its logo in a context that has not been preapproved by the company’s PR office.
What happens as governments themselves awaken to the idea of preemptively stopping all private communications they dislike?
It would take but a simple court order for a country to forcibly compel Facebook to add an additional set of content filters for its citizens to ban them from sending or receiving messages disliked by that government. A repressive regime could ban all conversation about democracy or rights and all criticism of the government. A country where being LBGT carries the death sentence could ban all mentions of LBGT culture.
A government working with Facebook to ban all terrorist content within its borders could easily utilize its same national security laws to force the company to ban all pro-democracy messaging it views as a threat to its existence.
Perhaps most troubling of all, however, is a question raised by one of Facebook’s engineers in the video. She notes that when content moderation is performed directly on users’ devices without any data being transferred to Facebook, the company has no way of knowing when violations occur or what the violating content was. If a user attempts to send a rules-violating post via encrypted WhatsApp message, the on-phone AI content moderation algorithm would flag the post and prevent it from being sent. However, Facebook itself would not have a record that the particular user attempted to post a piece of violating content and won’t have a copy of that piece of content to help fine-tune its algorithms over time.
The engineer raises the question of how Facebook might receive alerts of attempts to post banned messages and how it could receive a copy of that content but moves on without answering that question.
This raises the troubling specter that Facebook’s on-phone moderation algorithms might ultimately be designed to send an alert back to the company every time they block a piece of content, along with a copy of the offending message.
Making matters worse, earlier this year the company clarified that it views any user that installs its app on their phone as granting the company the legal right not only to track their realtime location through their phone’s GPS, but more disturbingly, to access their camera, microphone and non-uploaded photos residing on the phone for any purpose.
The terms governing the company’s access to our phones are extraordinarily broad, with the company specifically calling attention to this line in its Data Policy that it has the right to access “information you allow us to receive through device settings you turn on, such as access to your GPS location, camera or photos” and use that information to “to verify accounts and activity, combat harmful conduct, detect and prevent spam and other bad experiences, maintain the integrity of our Products, and promote safety and security on and off of Facebook Products … [including] investigate suspicious activity or violations of our terms or policies, or to detect when someone needs help.”
Facebook acknowledged earlier this year to secretly tracking the realtime locations of users it deems a threat to the company without their knowledge or permission.
It is not hard to imagine the company going a step further and turning its two billion users’ devices into a global surveillance network beyond the wildest dreams of the world’s most repressive governments.
Edge AI is the key to that Orwellian vision.
Today if Facebook wanted to scan all of the photos on a user’s phone to see if they’ve ever taken photos of Facebook properties or posed with a weapon or if they mention Facebook in their voice phone calls, Facebook would have to upload all of that data back to its servers, which would likely violate any number of wiretapping laws, not to mention saturate the user’s mobile data quota.
Instead, once Facebook’s on-phone content scanning algorithms are sufficiently robust, it no longer has to ship anything off of the device. Image recognition algorithms can scour the user’s private photo gallery and monitor every photo they take, including the ones they never share with anyone. Voice recognition algorithms can monitor their phone calls and flag every mention of Facebook and the context it is mentioned in. The microphone could even be left permanently on to scan the surrounding background environment 24/7, creating a globally distributed network of billions of always-on microphones transcribing global private conversations.
In many ways, Facebook’s shift towards a “privacy first” encrypted future might better be described as “moving Facebook’s surveillance to the edge.”
The company did not respond to several emails requesting comment.
Putting this all together, this is not some imaginary science fiction dystopia of the faraway future. It is the very real Orwellian world Facebook is bringing to us today. The underlying technologies are all here and as the company’s F8 presentation vividly illustrates, it is investing heavily and making rapid strides towards this future.
In the end, perhaps the transition towards a “privacy first” future was actually Newspeak for our first step towards 1984.
