Facebook Announces Messenger Security Features that Don't Compromise Privacy

Note that this is "announced," so we don't know when it's actually going to be implemented.

Facebook today announced new features for Messenger that will alert you when messages appear to come from financial scammers or potential child abusers, displaying warnings in the Messenger app that provide tips and suggest you block the offenders. The feature, which Facebook started rolling out on Android in March and is now bringing to iOS, uses machine learning analysis of communications across Facebook Messenger's billion-plus users to identify shady behaviors. But crucially, Facebook says that the detection will occur only based on metadata­ -- not analysis of the content of messages­ -- so that it doesn't undermine the end-to-end encryption that Messenger offers in its Secret Conversations feature. Facebook has said it will eventually roll out that end-to-end encryption to all Messenger chats by default.

That default Messenger encryption will take years to implement.


Facebook hasn't revealed many details about how its machine-learning abuse detection tricks will work. But a Facebook spokesperson tells WIRED the detection mechanisms are based on metadata alone: who is talking to whom, when they send messages, with what frequency, and other attributes of the relevant accounts -- essentially everything other than the content of communications, which Facebook's servers can't access when those messages are encrypted. "We can get pretty good signals that we can develop through machine learning models, which will obviously improve over time," a Facebook spokesperson told WIRED in a phone call. They declined to share more details in part because the company says it doesn't want to inadvertently help bad actors circumvent its safeguards.

The company's blog post offers the example of an adult sending messages or friend requests to a large number of minors as one case where its behavioral detection mechanisms can spot a likely abuser. In other cases, Facebook says, it will weigh a lack of connections between two people's social graphs -- a sign that they don't know each other -- or consider previous instances where users reported or blocked a someone as a clue that they're up to something shady.

One screenshot from Facebook, for instance, shows an alert that asks if a message recipient knows a potential scammer. If they say no, the alert suggests blocking the sender, and offers tips about never sending money to a stranger. In another example, the app detects that someone is using a name and profile photo to impersonate the recipient's friend. An alert then shows the impersonator's and real friend's profiles side-by-side, suggesting that the user block the fraudster.

Details from Facebook

EDITED TO ADD (7/1): This entry has been translated into Spanish.

Posted on May 29, 2020 at 6:37 AM • 6 Comments


GatoMay 29, 2020 6:55 AM

If Facebook's other attempts in this area are any indication, it will be a dismal failure. Their machine learning (I refuse to call it AI) can't even locate and shut down fraudsters openly congregating and discussing fraud when they can read the actual message contents. It is a pipe dream to think they can do so with only metadata. Machine learning is decades away from being able to do this. So long as facebook tries to police billions of humans with a few thousands of human moderators and ineffective machine learing "big data" tools the result will continue to be failure.

NameMay 29, 2020 7:12 AM

I mean,
how will this not be abused the same way Google has suppressed cryptocurrencies? Fraud and child abuse sounds so fishy in itself. "Oh please, wont somebody think of the children?"

AlejandroMay 29, 2020 10:21 AM

Why do I assume this is nothing more than another data grabbing scam, again?

Patrick HuttonMay 30, 2020 5:49 PM

Judging by Facebook's announcement metadata isn't being encrypted. No doubt it'll have a convenient message backup in clear text too.

Not great security.

metaJune 1, 2020 10:58 PM

Bruce, that's a misleading title/headline!
“Facebook Announces Messenger Security Features that Don't Compromise Privacy”
but metadata compomises privacy, should the title should be something like
“Facebook Analysing Messenger Metadata; Says It Will Help Users Stay Safe”
Presumably it's getting a lot of information from the metadata, and has now announced an intention to share just some of this with users.

Contrast this with Signal's approach, minimising availability of such metadata to the service: https://signal.org/blog/sealed-sender/

MarkJune 2, 2020 9:02 PM

You know which adults "send messages or friend requests to a large number of minors"?


I predict that if this ever gets rolled out, there'll be a huge uproar over Facebook falsely accusing people of being child molesters.

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Sidebar photo of Bruce Schneier by Joe MacInnis.