Until the digital era, communications technologies in the modern world were largely content-neutral. The USPS did not systematically open every letter and review its contents to decide whether to deliver it, the phone company did not listen in on our calls and disconnect them at the first sign of criticism of elected officials and restaurants, bars, coffee shops and parks did not install microphone networks to listen in for thoughtcrimes. In the absence of technologies that could make such mass surveillance tractable we never even contemplated the ramifications of all-seeing omnipresent government monitoring. Now that society stands on the cusp of such tools becoming available, what might this future dystopia look like?
At Facebook’s F8 conference last week, the company touted its move to the edge and showcased the results of its initial experiments in what it might look like to move all of its content moderation algorithms out of the data center and onto user’s own phones. While Facebook’s efforts are still at their earliest stages, the underlying technologies to run complex AI content recognition algorithms directly on smartphones is already here.
As the idea of performing content moderation directly on users’ phones matures, what would a world look like in which content analysis becomes scalable to the level of an entire society?
The telephone company cannot afford to pay an army of human reviewers to listen in to every phone call its customers make each day.
AI suffers no such limitations, but the costs of deploying that much hardware to run all of those AI algorithms would be staggering. Instead, by offloading the AI processing to users’ phones, content analysis can scale linearly to the entire population.
All of a sudden, phone companies could use their customers’ own phones to monitor their calls, transcribing every call in realtime and flagging any word that might indicate a thoughtcrime.
End-to-end encryption would be of little help when the scanning is occurring on users’ devices before and after encryption.
Even more troubling, once the idea of realtime transcription and content scanning of phone calls becomes normalized, it would not take very much to extend that a step forward to monitor the phone’s microphone 24/7, rather than only during calls.
Every phone’s microphone could be left on continuously and processed by the same transcription algorithms, listening in on the background audio around each person to flag any discussion that veers into “unacceptable speech.”
In such a world it would not matter that two parties take every precaution to leave all of their electronic devices at home before meeting in the back of a coffee shop. All it would take is a single person elsewhere in the room with their phone for an algorithm to hone in on that clandestine meeting and alert authorities.
One could even imagine a system that flagged all conversations by individuals who did not bring their phones with them or who turned their phones off prior to their meeting, as evidence for the government that they have something to hide.
Every house could be required to have at least one phone present and powered at all times to monitor conversations within its walls, much as 1984’s world was saturated with telescreens.
In turn, as our written conversations increasingly occur over digital mediums and as our digital exhaust betrays our interests, even our unspoken thoughts are at risk.
Putting this all together, the technology that could enable this kind of pervasive surveillance, turning every phone into an always-on monitoring device, is already here. As social media companies begin to normalize its deployment and its accuracy continues to improve, it will only be a matter of time before there will be no such thing as a “private” conversation.
