Facebook engineering manager Mark Harman, who also co-founded the Sapienz bug hunting system,
To keep Facebook humming along for its 2 billion users around the world, it has a lot of software engineers looking for a lot of bugs. Over the course of a week, they’ll typically make around 100,000 changes to Facebook’s source code, which also receives 1 million different commands everyday. Many will tell you that finding the bugs and fixing them is dull, time-consuming work. Increasingly at Facebook, it’s getting automated.
The company is gradually rolling out an automated tool called Sapienz that can find those bugs faster than human developers can, and suggest fixes based on past ones made by engineers. So far, it’s being used on the main Facebook app, Instagram, Messenger and enterprise offering Workplace, though not WhatsApp.
Finding bugs can sometimes take days or even weeks, as programmers manually generate tests that mimic what human users might do on an app. But developers who use Sapienz are finding bugs 17 times faster than they normally would because software pretending to be a human is stumbling across those bugs instead, according to Facebook engineering manager Mark Harman, who gave a talk to journalists about the tool in London earlier this week. Harman is a former academic who developed Sapienz with two other computer scientists.
The trio were acqui-hired by Facebook in 2017. “I did feel some trepidation,” Harman said of coming to work for the world’s biggest social network. “But I dove in. What better opportunity to work at scale.”
Sapienz was the subject of a PhD of Harman’s co-founder, and was deployed to Facebook’s platform after 17 months. “It’s been running for a year in the background,” said Harman. “When we first deployed it was just on the main Facebook app but we’re rolling it out wider.”
“We’re getting 75% of bugs fixed before they get into the product,” he added. “It’s like a helping hand for the the developer. Anecdotally we know it’s making it a lot faster.”
The idea of automating the search for bugs isn’t new, but Harman says Sapienz brings together three types of expertise in the field: specifying where a bug is, reducing the scale of test sequences and pinpointing a bug itself.
Harman said it’s important to still include humans in the process of finding and fixing live software, for the sake of accountability. But a bug-hunting tool like Facebook’s can also help make programming a more creative process, by replacing “the nitty-gritty stuff that puts people off.”
Whether humans should have greater exposure to coding flaws is something of a philosophical debate in computer science, and Harman compares it to the impact of automated spellcheckers. “Does it make your spelling worse,” he asks, and does it matter if it gets worse?
Harman said Facebook has plans to eventually make Sapienz open source. Would Facebook will ever pivot the Sapienz engine for mimicking human behavior on to another use case? Harman doesn’t know, but Sapienz’ underlying concept of using “computational search” has been applied to design too, he says. "Humans can’t detect all bugs."
