Schneier on Security

mostly harmful • November 10, 2017 12:56 PM

@alejandro “Why would anyone in their right mind GIVE FB nude pics???”

“Mark” Mountain-of-Suckers doesn’t know the answer either: http://www.businessinsider.com/well-these-new-zuckerberg-ims-wont-help-facebooks-privacy-problems-2010-5

Zuck: Yeah so if you ever need info about anyone at Harvard

Zuck: Just ask

Zuck: I have over 4,000 emails, pictures, addresses, SNS

[Redacted Friend’s Name]: What? How’d you manage that one?

Zuck: People just submitted it.

Zuck: I don’t know why.

Zuck: They “trust me”

Zuck: Dumb fucks

Say what you will, it’s no lie.

mostly harmful • November 10, 2017 6:28 PM

So, about this photoDNA project in general, and considered apart from Facebook’s prophylactic use-case contra revenge porn: Having read the descriptions linked above, I wondered about photoDNA’s potential for false positives. The obvious search terms yielded no github repo or anything inspectable like that. (Also, I did check out Hany Farid’s faculty page at Dartmouth, which looks very interesting, but AFAICT all of the photoDNA material linked there was popular press: http://www.cs.dartmouth.edu/farid/ )

That was kind of disappointing. And kind of disturbing, too, given a project whose promoters (whom I herein assume are disjoint from the developers) so breathlessly try to convince casual readers is vewy vewy important and moreover ready to roll.

Some popular-oriented articles from a search engine did mention the false positive concern briefly, as did a 2011 New York Times article linked on Farid’s website:

https://gadgetwise.blogs.nytimes.com/2011/05/19/facebook-to-combat-child-porn-using-microsofts-technology/

PhotoDNA works by creating a “hash,” or digital code, to represent a given image and find instances of it within large data sets, much as antivirus software does for malicious programs. However, PhotoDNA’s “robust hashes” are able to find images even if they have been altered significantly. Tests on Microsoft properties showed it accurately identifies images 99.7 percent of the time and sets off a false alarm only once in every 2 billion images, and most of them point to nearly identical images, Dr. Farid said.

http://www.securityfocus.com/news/11570

“We tested [the PhotoDNA tool] over billions and billions of images,” he said. “We tried very hard to make it very efficient … and to minimize the false alarm rate.”

The securityfocus article does not test its—doubtlessly deeply concerned—readers’ patience with any further discussion of what it means to “tr[y] hard […] to minimize the false alarm rate.”

But, okay, so they collected and checked a set of billions of images (on Microsoft’s premises, I gather) for false positives. Fine, so far as that goes. But how far does that go?

Hard to say. Notably, I found no mention of the developers trying to construct their own false positives. Have they tried to do so at all? (We can be nearly certain that the developers did not attempt to construct billions of them; the two-billion-to-one result is from testing whatever collection of images Microsoft happened to host at the time.)

In the absence of evidence of any serious attempt to do so, and given the potentially dire consequences for malicious construction of false positives (and failure to detect the same), I find the fanfare premature. Suspicious, even.

We already have quite a long history of forensics professionals misrepresenting, in the courtroom, the potential for false positives regarding other sorts of “fingerprints”, with tragic consequence for the wrongly accused. I’d be happy to discover that a systemic lesson has been learned, but I am not going to hold my breath for that.

In this regard, do note the suggestive language employed by that 2011 article in the New York Times [bold mine]:

Microsoft says it has refined a technology it created called PhotoDNA to identify the worst of these disturbing images — even if they are cropped or otherwise altered — and cull through large amounts of data quickly and accurately enough to police the world’s largest online services.

Note as well, in that same NYT piece, the disturbingly familiar tune sung by the Mouth of Sauro—cough—by Microsoft’s general counsel, one Mr Brad Smith:

“We’re very passionate about PhotoDNA because we’ve seen it work,” said Brad Smith, Microsoft’s general counsel. “We invented it through Microsoft research, and we are trying to give it away free, including to our competitors.”

He encouraged consumers to pressure online services to adopt it.

I would be grateful if anyone can share a pointer to more information regarding my primary concern, namely photoDNA’s resistance to malicious construction of false positives. I find the technical topic intriguing, but am totally bored by the breathless soccer-mom fearmongering that threatens to consume every last column-inch.

mostly harmful • November 14, 2017 4:23 AM

I wrote above, regarding the oversell of photoDNA hype:

We already have quite a long history of forensics professionals misrepresenting, in the courtroom, the potential for false positives regarding other sorts of “fingerprints”, with tragic consequence for the wrongly accused. I’d be happy to discover that a systemic lesson has been learned, but I am not going to hold my breath for that.

Regarding that long history (h/t @BarrettBrown_):

FBI’s flawed forensics expert testimony: Hair analysis, bite marks, fingerprints, arson; OR
The FBI faked an entire field of forensic science
by Dahlia Lithwick, in Slate 2015-04-22
https://www.slate.com/articles/news_and_politics/jurisprudence/2015/04/fbi_s_flawed_forensics_expert_testimony_hair_analysis_bite_marks_fingerprints.html

FBI admits flaws in hair analysis over decades
by Spencer S Hsu, in the Washington Post 2015-04-18
https://www.washingtonpost.com/local/crime/fbi-overstated-forensic-hair-matches-in-nearly-all-criminal-trials-for-decades/