Developments in Facial Recognition

Eventually, it will work. You'll be able to wear a camera that will automatically recognize someone walking towards you, and a earpiece that will relay who that person is and maybe something about him. None of the technologies required to make this work are hard; it's just a matter of getting the error rate down low enough for it to be a useful system. And there have been a number of recent research results and news stories that illustrate what this new world might look like.

The police want this sort of system. I already blogged about MORIS, an iris-scanning technology that several police forces in the U.S. are using. The next step is the face-scanning glasses that the Brazilian police claim they will be wearing at the 2014 World Cup.

A small camera fitted to the glasses can capture 400 facial images per second and send them to a central computer database storing up to 13 million faces.

The system can compare biometric data at 46,000 points on a face and will immediately signal any matches to known criminals or people wanted by police.

In the future, this sort of thing won't be limited to the police. Facebook has recently embarked on a major photo tagging project, and already has the largest collection of identified photographs in the world outside of a government. Researchers at Carnegie Mellon University have combined the public part of that database with a camera and face-recognition software to identify students on campus. (The paper fully describing their work is under review and not online yet, but slides describing the results can be found here.)

Of course, there are false positives -- as there are with any system like this. That's not a big deal if the application is a billboard with face-recognition serving different ads depending on the gender and age -- and eventually the identity -- of the person looking at it, but is more problematic if the application is a legal one.

In Boston, someone erroneously had his driver's licence revoked:

It turned out Gass was flagged because he looks like another driver, not because his image was being used to create a fake identity. His driving privileges were returned but, he alleges in a lawsuit, only after 10 days of bureaucratic wrangling to prove he is who he says he is.

And apparently, he has company. Last year, the facial recognition system picked out more than 1,000 cases that resulted in State Police investigations, officials say. And some of those people are guilty of nothing more than looking like someone else. Not all go through the long process that Gass says he endured, but each must visit the Registry with proof of their identity.


At least 34 states are using such systems. They help authorities verify a person’s claimed identity and track down people who have multiple licenses under different aliases, such as underage people wanting to buy alcohol, people with previous license suspensions, and people with criminal records trying to evade the law.

The problem is less with the system, and more with the guilty-until-proven-innocent way in which the system is used.

Kaprielian said the Registry gives drivers enough time to respond to the suspension letters and that it is the individual’s "burden’" to clear up any confusion. She added that protecting the public far outweighs any inconvenience Gass or anyone else might experience.

"A driver’s license is not a matter of civil rights. It's not a right. It’s a privilege," she said. "Yes, it is an inconvenience [to have to clear your name], but lots of people have their identities stolen, and that’s an inconvenience, too."

IEEE Spectrum and The Economist have published similar articles.

EDITED TO ADD (8/3): Here's a system embedded in a pair of glasses that automatically analyzes and relays micro-facial expressions. The goal is to help autistic people who have trouble reading emotions, but you could easily imagine this sort of thing becoming common. And what happens when we start relying on these computerized systems and ignoring our own intuition?

EDITED TO ADD: CV Dazzle is camouflage from face detection.

Posted on August 2, 2011 at 1:33 PM • 53 Comments


Richard Steven HackAugust 2, 2011 1:52 PM

Speaking of "looking like someone else", I used to comment to cons in the joint about their tattoos: "Nothing like telegraphing to the cops that you're an ex-con. Can't wait until you get arrested for just having a tattoo similar to the one that bank robber a few blocks over had...Might as well have a bulls-eye tattooed on your forehead..."

I talked to one con who had a record of robbery, but had never robbed a bank. He was nonetheless arrested on a bank robbery charge by a cop who wanted to nail him. He was convicted based on the fact that the prosecutor had him show his tattoos in open court...

I was just reading today a Web site that had a piece on evading facial recognition by wearing makeup in what are similar to camouflage patterns which alter the contrast in the face.

Ah, here it is:

How to camouflage yourself from facial recognition technology

One point the cyberpunk RPG manuals (as we discussed in an earlier thread) point out is that a lot of the time street cameras won't be picking you up at the correct angle to apply facial recognition. In the movies, no matter the angle or how blurry the shot, they always manage to get a face match in seconds. In the real world, it undoubtedly won't be so easy.

The RPG games also recommend wearing hard masks on a "shadowrun" to evade software that can reconstruct facial contours even through a soft mask like a balaclava or nylon stocking.

Using lasers to blind the cameras is another trick. Just saw that used in a recent episode of the Canadian TV series "XIII The Series".

Of course, none of these tricks work if you're just trying to get into a building and the front door has one of these things pointing right at you - along with a guard.

Then it's time to use a front man unconnected with you to run your errands... :-)

Glenn MaynardAugust 2, 2011 1:58 PM

A driver's license is a requirement to live in modern society. It must be treated as a right, only to be revoked through due process, and not as a privilege that the bureaucracy can revoke at its whim--regardless of their rationalization.

The government can't throw someone in prison without due process on the handwaving rationalization of "protecting the public". It shouldn't be allowed to revoke a person's ability to commute, hold a job, pay their mortgage and feed their children without due process, either.

NotEvenCloseAugust 2, 2011 2:00 PM

How do you prove your identity if you don't have a driver's license? That's the de facto identity card now.

-BAugust 2, 2011 2:21 PM

"...protecting the public far outweighs any inconvenience Gass or anyone else might experience."


Oliver HollowayAugust 2, 2011 2:46 PM

Sufficiently-accurate facial recognition software has been "around the corner" for a long time. Maybe someday it will be sufficiently accurate, but until then, your identity is at risk due to mis-use by the authorities.

PaulAugust 2, 2011 2:54 PM

"Yes, it is an inconvenience [to have to clear your name], but lots of people have their identities stolen, and that’s an inconvenience, too."

Identity theft is a crime. Is the State justifying its actions based on the fact that criminals behave similarly? I guess this explains tax collection.

Nitpicker #753August 2, 2011 3:00 PM

Loudspeaker/earpiece, not microphone. You have no idea how confused I got reading this, skimming back and forth trying to figure out what the microphone was for. ;-)

Petréa MitchellAugust 2, 2011 3:01 PM

The Boston Globe article says 1000 cases resulting in an investigation per year, but 1500 suspension letters sent out every day. Presuming that's only per weekday, that still works out to roughly 390,000 suspensions per year (6% of the total population of Massachusetts!), or a false positive rate of 99.7%, before you even count the people who were investigated but cleared.

...I think the reporter must have gotten that suspensions-per-day number wrong.

AdamAugust 2, 2011 3:05 PM

A driver's license isn't a requirement to live in modern society. I gather that plenty of inhabitants of cities like London and New York don't have them. I don't have one myself and it's not been a problem so far.

Glenn MaynardAugust 2, 2011 3:33 PM

Uh, not everyone lives in New York and London. I live a fifteen minute drive from the nearest train and an hour from the nearest major city.

GeorgeAugust 2, 2011 3:54 PM

Rachel Kaprielian, the head of the Massachusetts Registry of Motor Vehicles, clearly deserves to join the ranks of the permanent unemployed. First, she asserts that the benefits of an imperfect facial recognition system somehow justify "burdening" the innocent victims of its imperfection with jumping through arbitrary bureaucratic hoops to clear their names.

Then she equates false positives with identity theft. She goes on to minimize the gravity of that offense to its victims by dismissing it as "an inconvenience," before implying that even identity theft is OK if it's the government that's perpetrating the offense in the name of "protecting the public."

The worst thing is that she's probably not even aware of what she's telling reporters. She seems to be a bureaucrat trying to cover her agency's dirty posterior by justifying a serious flaw in the system, and then trying to blame the victims when results in something inexcusable.

Ms. Kaprielian need not worry, though. If she is booted out of her job as she deserves, I'm sure John Pistole has a senior management position waiting for her. Her arrogance, her contempt for the public and the rule of law, and her attitude that any injustice, incompetence, or abuse is always justified when an agency is "protecting the public" all make her a perfect fit for the TSA.

antonAugust 2, 2011 4:05 PM

"The problem is less with the system, and more with the guilty-until-proven-innocent way in which the system is used."

Thanks Bruce for this. Needs to be said more and more as this kind of mentality creeps in insidiously more and more from from every direction.

EricAugust 2, 2011 4:16 PM

Where I work we tried facial recognition and had so many problems with it, that we gave it up. There is a major problem when you decide to do something "nasty" (messing up a trip, etc.) to an innocent person. This is something that the lawyers must love.

anonymousAugust 2, 2011 4:45 PM

The problem is less with the system, and more with the guilty-until-proven-innocent way in which the system is used.

Additionally, the problem also seems to be the base rate fallacy.

Dr. TAugust 2, 2011 5:47 PM

"... Facebook has recently embarked on a major photo tagging project, and already have the largest collection of identified photographs in the world outside of a government...."

Do the execs at Facebook realize that a significant percentage of customers do not use their own photos in their profiles?

KennAugust 2, 2011 6:40 PM

someone may not have a picture of themselves on their own profile, but i'd be willing to bet their friends do. and then those friends just have to tag that picture with the user's name.

pfoggAugust 2, 2011 7:38 PM

"A driver’s license is not a matter of civil rights. It's not a right. It’s a privilege,"

*Driving* is a privilege, not a right, and the onus is on drivers to establish, and re-establish on reasonable demand, their qualifications as a legal driver.

If we overload civil legal identity onto driver's licenses, then I think you can make a demand for constitutionally guaranteed due process before revoking it.

I don't think state ID cards (for non-drivers) can be held if a driver's license has been issued, so revoking the driver's license revokes verifiable personal ID, unless a passport is being overloaded as a general ID.

Of course, if we clearly establish (by law or precedent, or common practice) that going anonymous is acceptable for all normal daily activities, then this is no longer a constitutional issue.

Richard Steven HackAugust 2, 2011 7:57 PM

The really stupid thing here in California is that if you have an expired Non-Driver's ID (as I do), when you go to replace it, it will take two months to get the new one with your picture on it. For those two months, all you have is the RECEIPT from the DMV that shows the license number, no picture, and the date..

Yet THAT is more accepted as "valid ID" than an EXPIRED Non-Driver's ID with a (previously valid) license number AND a picture!

Two seconds of thought will show that apparently the only thing needed to be "valid" is the fact that the piece of paper is not "expired" - which is a totally arbitrary and meaningless phenomena related to how long an ID is allowed to exist before being replaced. Which has nothing whatever to do with "identity".

In other words, my actual picture is worthless and even the license number is not that important compared to the mere expiration date of the license.

Some idiots claim what is important is that the ID "links" to you. But exactly how? No one verifies the address on it - it could be faked. None of the information on the license actually enables anyone to FIND you. The ONLY real identification an ID provides is linking the picture to a name and a number. And really only the picture is what one would think of as "identification" since the other two could be faked.

But the receipt doesn't do that at all. Yet it is more "valid" than an expired ID with a picture, a name and the number.

It's ridiculous.

I haven't bothered to renew my Non-Driver's ID because even my bank no longer requires a picture ID, just my ATM card and PIN. The only place I need the Non-Driver's ID is in some client's apartment building (and the Federal Building if I ever need to go in there.)

nobodyspecialAugust 2, 2011 10:19 PM

Where can I buy one?

A 400fps camera with high enough resolution to capture 46,000 features on a face 50m away would cost around $150K from Phantom and need a serious data capture system

One that will fit in your glasses would be amazing

bkd69August 2, 2011 11:15 PM

I realize it hardly needs to be said among this lot, yet I'll say it anyway...

Isn't one of the founding ideals of our justice system that we tolerate some level of criminal freedom just so that we don't inadvertently punish the innocent?

This could cost the state severely in cash, if some lawyer and plaintiff/defendant wanted to make some hay. If I were a lawyer, I'd be arguing that until the state has sufficient evidence to file charges and make an arrest, they have no business suspending licenses or invalidating IDs.

NobodyspecialAugust 2, 2011 11:37 PM

@bkd69 - would that same arguement work for those people who are suspected terrorists dangerous enough not to be allowed to fly, but not dangerous enough to even question when you turn them away at the check-in desk?

nemikAugust 2, 2011 11:52 PM

But we have to inconvenience the public in order to properly prevent the public from being inconvenienced.

StanAugust 3, 2011 12:25 AM

Let's accept the argument that a drivers license is a privilege. If the government accuses you of something, such that it revokes your privileges, then you are being punished and as such need due process.

So who should have the burden, the government with very deep pockets or the falsely accused who are unfairly subsidizing the govt. by working and paying to clear up a mistake made by the govt.? It seems to add insult to injury, falsely accused? No problem work and pay to have it resolved.

Dirk PraetAugust 3, 2011 2:53 AM

@ George

I have it on good authority that Ms. Kaprielian is actually a Chinese foreign exchange official whose real name is Sue Yu Tu. This information was recently revealed when Rupert Murdoch had contacted her to replace Rebekah Brooks as head of News International.

"lots of people have their identities stolen, and that’s an inconvenience, too."

So the rationale seems to be that it's perfectly ok for governments to go down the same path as gangsters and criminal organisations but that this is not a problem since they are the good guys ? No wonder there are groups such as Anonymous.

*tinfoil hat*
Can someone please send me a Guy Fawkes mask before in some Kafkaian MFU I am being sent a government Terminator because of my resemblance to Indian actor Sanjay Dutt who landed in jail because of suspected terrorist ties ?
*tinfoil hat*

JohnAugust 3, 2011 3:52 AM

people have been able to recognise an extensive range of other people for a significant time now so obvious people are the problem and should be banned.

Mike JAugust 3, 2011 3:53 AM

Don't forget the system that was installed for the Beijing Olympics claimed very good facial recognition technology and tracking of individuals over very large areas, multiple cameras.
These companies had input apparently.

We may never know how well it worked, but at least it has been attempted once.

( I have a horrible memory for names, so if my phone can tell me who i am talking to I wouldn't mind all that much)

A BicyclistAugust 3, 2011 4:26 AM

@ Glenn Maynard

A driver's license is a permission to operate heavy machinery in public, shared space, according to a set of rules. Violating these rules can result in severe injury or death, often of other parties. Because of this, a driver's license should be regarded as a privilege that can be taken away rather than a right due to all.

Its use as a form of ID is function creep, just like using SS Number as a general identifier for people. I'm sure Bruce has written about the problems with this.

Thomas B.August 3, 2011 5:37 AM

"the prosecutor had him show his tattoos in open court"

This is not typical, the defense should have objected.

I worked in a prosecutor's office. Defendants often had make up artists obscure their tattoos in the morning before trial to avoid prejudicing the jury. It was amazingly careful and subtle work.

JonathanAugust 3, 2011 5:47 AM

I've always wondered about the state of ID in the USA, and what people who choose not to drive do.

Here in Israel, every citizen is issued an ID number upon birth (or during some stage of the immigration process), and at age 16 your get a photo ID card with that number on it. This ID number is used in a variety of other cards - driver's license, student license, even some supermarket affinity cards. Most places will accept a driver's license (which includes your photo) as an ID, but the only official ID is the state-issued ID card.

There is the problem that these ID cards are easily forgable using a color laser printer and a laminator...

Thomas B.August 3, 2011 5:53 AM

> It is said that there is a doppelganger for everyone.

Think of all the fun games you can play with the system once you find a few face recognition doubles.

I think the biometrics literature uses animal analogies to break down the false positives in biometrics. Sheep are most of the people in the system. Goats are people who are hard to identify, and might not even resemble themselves from one day to the next. Lambs seem to all resemble each other, and are easy to imitate. Wolves are people who are great at imitating others.

It's true that the system could gradually get lower overall error rates, but you hit trade-offs at a certain point, where decreasing the number of lambs increases the number of goats. There are a series of errors pulling the system in different directions.

These errors are predictable enough in the population to be exploitable by deliberate attackers.

JeffAugust 3, 2011 7:35 AM

"protecting the public"? A joke. The purpose is to bilk more money out of the public.

Richard Steven HackAugust 3, 2011 8:38 AM

Jonathan: In the US, licenses are a little harder to forge due to various technology tricks like holograms and embedded fibers - but not much. There is a brisk business in doing so, and even college students get into the game, as underage drinking is a big deal so students need to purchase fake IDs to get into bars.

Then there's the immigration problem, with millions of illegal immigrants needing fake ID.

So there's a huge underground business in fake IDs in the US. In some cases, the people running these businesses have confederates inside the Department of Motor Vehicles in the states in which they operate who procure official forms and also allow the data to be entered into the computer systems, thus making the ID better than merely a forged one.

I've read that in LA you just need to go into the Hispanic neighborhoods and people will approach you in your car offering forged documents. You give them a photo and in a few hours and a few hundred dollars you have a very good ID.

So as usual, in the end, it's the average consumer who has to put up with a two month wait for a "legit" ID, while any criminal willing to spend a few hundred bucks can get a usable ID in a matter of hours.

Black markets are almost always more efficient than legal markets.

shAugust 3, 2011 10:17 AM


I'll agree that they're probably exaggerating their tech's abilities, but I think you're wrong that the glasses' camera is completely unrealistic. It said 400 facial images per second, but that doesn't have to translate into 400 frames per second. You only need about 14 faces per frame to get 400 faces in 30 frames.

As to resolution, I'm not sure what constitutes a facial "feature", but if you can fit 46,000 of them in one face, I'm going to assume they're pretty tiny. If we guess a feature can be represented with a 5x5 pixel square (I'm thinking that's probably generous), that comes to about 1.2 megapixels per face, implying a 17 megapixel image to hold about 14 faces.

17MP @ 30fps doesn't sound terribly unrealistic to me.

uk visaAugust 3, 2011 11:46 AM

@George I completely agree that Ms Kaprielian should lose her job because of her unilateral decision that the presumption of innocence doesn't fit her plans for the cleansing of Massachusetts. Even if we're wrong, surely her getting another job is nothing more than a minor inconvenience.
Equally, it's a sad state of democracy that neither the Registry nor State Police saw fit to keep tabs on the number of people wrongly tagged by the system; such oversight is tantamount to negligence.

mndeanAugust 3, 2011 11:50 AM

Do the execs at Facebook realize that a significant percentage of customers do not use their own photos in their profiles?

Most people I know don't. If facebook thinks that's me, then I'm an actor who's been dead decades.

paulAugust 3, 2011 1:45 PM

What does someone want to bet that no suspension letter has ever gone out to an elected official of a state, or to a management-level state employee or a member of the state's law enforcement contingent?

Richard Steven HackAugust 3, 2011 2:38 PM

From the article referenced by DLM above:

"“About 89 percent of genuine driver’s licenses don’t comply with their own specifications 100 percent,” Williams said.

Intellicheck feeds the imperfections into its database. A license perfect in every respect immediately reveals itself as a fraud when compared with the flawed real ones."

Anyone notice the irony here? If people knew how to check a driver's license, 89 percent of car owners would be in jail for identity theft?

GeorgeAugust 3, 2011 3:17 PM

@bkd69: "Isn't one of the founding ideals of our justice system that we tolerate some level of criminal freedom just so that we don't inadvertently punish the innocent?"

Yes, that was one of the founding ideals. But 9/11 changed everything, and made all the founding ideals obsolete. Better to punish thousands by mistake and let them prove their innocence than for one government bureaucrat to be blamed and lose their job after failing to prevent a terrorist attack. The 9/11 victims wouldn't want it any other way.

In Schneier We TrustAugust 3, 2011 4:43 PM

What a scary, completely public and digitized world this is becoming...

Can anyone besides me come up with a bazillion ways this technology will be abused..?

pfsmAugust 3, 2011 4:51 PM

One of my Facebook friends came up with a cool idea for a Christmas card: he's a collector of seashells, and made a photograph of a sand pile shaped similarly to a Christmas tree and studded with shells, then tagging all his friends and sending them the pic. Similarly about half the photos tagged on facebook as me are pictures of license plates. Guess what my hobby is. What I'm saying is that facebook's idea of facial identity from photograph tags is not without problems.

Richard Steven HackAugust 3, 2011 9:57 PM

If someone tagged my name to every picture I'm involved with on the Net, they would think I'm Summer Glau or any of dozens of other hot babes.

In fact, I don't think there's a picture of me anywhere on the Net.

I just did a Google search - I'm right. The primary picture that comes up is movie/TV critic and biographical writer Richard Hack. But the only female pics that come up are Angelina Jolie and Diane Lane - and those two tag to me correctly.


Oh, wait, I didn't use Richard Steven Hack. That Google image search comes up with a number of hot babes. But the main image is my avatar - Thanos from Marvel Comics.

So now I'm either Angelina Jolie, Diane Lane, several other babes, or a comic book character.

RandomPersonAugust 4, 2011 9:16 AM

The license thing is scary. I'm one of those people who just doesn't look the same way IRL as I do in photographs.

GeorgeAugust 4, 2011 12:53 PM

RandomPerson, you have nothing to worry about.

Apparently the system compares driver's license photos. If it decides that two of them look alike, it immediately cancels both licenses, thereby "protecting the public." Individuals convicted of fraud by the system may be able to reinstate their licenses if they visit the Registry office and bring proof of identity sufficient to convince a human official that the system made a mistake and they are not guilty of fraud. But the burden of proof is on the convicted criminal to satisfy whatever standards of innocence and error that the human official decides are appropriate.

If you have a problem with that, there's nothing you can do. Rachel Kaprielian, acting under her unlimited authority to "protect the public," has determined that the benefits of this system entirely justify whatever "inconvenience" it might cause when it makes an error. In a press conference today, she emphatically reiterated that point, and added that "it's their own fault if they fail to maintain a facial appearance that sufficiently distinguishes their driver's license photographs from those of other drivers." Then she dismissed the reporters, stating that "I'm too busy protecting the public to waste my valuable time on ignorant dolts who insist on chasing phantoms of lost liberties."

TRXAugust 5, 2011 9:15 PM

Error rates are a problem... but I've encountered a number of unrelated people who could be twins. And if you call up any of several pictures of Hassan Nasrullah you'll see a face I can't distinguish from my own in the shaving mirror. You could put his face on my driver's license and not even the most diligent observer would question it.

If any system misidentified me as Hassan Nasrullah I couldn't blame the system or its programmers. We have the same face, for all practical purposes.

Unfortunately, good old Hassan is the head of the terrorist group Hezbollah. For some years now I've been expecting an armed SWAT response when I walk into the local Federal building...

A face isn't a unique ID token. Yes, you could refine your algorithms down until you could differentiate between Hassan and myself, but at that level of fineness, it would have to account for aging, not enough sleep the night before, or a three-martini lunch. The day-to-day variations in our faces are probably well within the differences between us.

wagolynnAugust 8, 2011 10:24 AM

Any such technology should have to automatically pay compensation to anyone incorrectly identified, say 1000 X the average wage. These would slowdown the deployment of these systems and perhaps encourage cross checking before applying any information derived from such systems.

JeremyAugust 15, 2011 7:22 AM

I found Kaprielian's comment "A driver’s license is not a matter of civil rights. It's not a right. It’s a privilege" very troubling - while it's true there are non-DLs, having a DL or non-DL is increasingly a requirement to vote in the US. And what could be more of a basic civil right than the right to vote?

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of Resilient Systems, Inc.