People Are Not Very Good at Matching Photographs to People

Professor Mike Burton, Sixth Century Chair in Psychology at the University of Aberdeen said: “Psychologists identified around a decade ago that in general people are not very good at matching a person to an image on a security document.

“Familiar faces trigger special processes in our brain—we would recognise a member of our family, a friend or a famous face within a crowd, in a multitude of guises, venues, angles or lighting conditions. But when it comes to identifying a stranger it’s another story.

“The question we asked was does this fundamental brain process that occurs have any real importance for situations such as controlling passport issuing and we found that it does.”

The ability of Australian passport officers, for whom accurate face matching is central to their job and vital to border security, was tested in the latest study, which involved researchers from the Universities of Aberdeen, York and New South Wales Australia.

In one test, passport officers had to decide whether or not a photograph of an individual presented on their computer screen matched the face of a person standing in front of their desk.

It was found that on 15% of trials the officers decided that the photograph on their screen matched the face of the person standing in front of them, when in fact, the photograph showed an entirely different person.

Tags: authentication, biometrics, identification

Posted on August 25, 2014 at 7:08 AM • 29 Comments

Comments

Andrew • August 25, 2014 7:35 AM

Maybe it’s time for two-factor identification on passports. When you register for your passport, you set a password/passphrase that is required when you present it at passport control. Storing it in the passport itself wouldn’t seem as secure, since it allows a forger to set their own password.

Bob S. • August 25, 2014 8:11 AM

I certainly would not count on pic vs. person ID for much of anything except general appearances, such as man or woman.

Women in particular with makeup and hair color changes can literally appear to be a different person altogether. With men, hats, glasses, even a shave or not can make a difference.

I realize there are computerized facial scans now. However, they are prone to error also. Many times they are counting on a non-smiling, straight on view for comparison.

(Unique characteristics are a whole other deal. Someone with one ear would be an easy match.)

dennis mccann • August 25, 2014 8:13 AM

Three dimensional or profile views added to the straight on shot could also help. We react emotionally to the person before us by instinct, and some comparative shots may help make the response more analytical. Also since animals seldom react to pictures unless trained Pavlovian-style, the two dimensional object on the page may not “map” to the person, so more variety of pictures may help create more points of comparison for our analytical side, not just our perception/visual side. An experiment mapping brian function to pictures vs people would be worth the investment ( using MRI to record brian regions stimulated)?

Iain Moffat • August 25, 2014 9:01 AM

I suspect two other factors in the effectiveness of human facial recognition are fatigue (can a passport officer be expected to be as diligent after 2 hours and 1000 passports as when he or she looks at the first one of the day ?) and stress (if there is a planeload of people crowding passport control most normal people will feel under pressure to process it quickly). This does seem like a case where a camera and computer should be there to support the human and say “Please check this one carefully because they don’t seem to match” even if all it does is compare the picture on the passport with the human in front of the desk in isolation.

Given that passports last 10 years and people have major changes in that timescale (I shaved my beard and had sufficient hair loss to look much older in a few years in my late 20s for example) I know that a simple match will often fail but I think humans will perform better if alerted to exceptions than if they have to remain alert when faced with a majority of correct matches,

Iain

Gavin B. • August 25, 2014 9:43 AM

This study should take into a similarity metric – how do the researchers specify an “entirely different person”.

Some face pairs are physical more similar than others – and so often they are confused by the observer.

Thus the result reported will depend on how the face-pair database was compiled. Which poupulation/s did it’s statistics characterise, etc?

It is hard to make definite conclusions without acknowledging a similiarity metric.

Scott "SFITCS" Ferguson • August 25, 2014 9:43 AM

@Iain Moffat

[snipped]

Please note – the original story was about Australian passports. I appreciate that you’ve probably missed the previous post explaining it and lacked the time to fully read the referenced story.
The biometrics of the face in the photo is encrypted into the passport. Which means that the previous posters “fear” it’ll be “forged” is, um, misplaced fear. The study was not a security study – but a study on facial recognition – of course that doesn’t draw the audience needed, that’s why we have journalists (sigh).

Given that passports last 10 years and people have major changes in that timescale (I shaved my beard and had sufficient hair loss to look much older in a few years in my late 20s for example) I know that a simple match will often fail but I think humans will perform better if alerted to exceptions than if they have to remain alert when faced with a majority of correct matches,

It’d be inappropriate for me to detail the exact facial biometrics used. But one commonly used in other countries is the “eyes to base of nose triangle ratio” which doesn’t change in adults. The most important thing to bear in mind when reading that, um, click-bait article is that it says nothing about passport inspection. It’s not secret that faces are scanned electronically and compared with the biometrics stored in the passport system – you don’t have to be face-on to allow accurate matching/recognition (in the order of tens of identifications per minutes from stills).

Recently had plastic surgery? Expect problems.

You may find it interesting to know that in Israel and some other countries your gait is used to uniquely identify you. “Human kinetics” is an important part of security processes. Did you know that a single scanned image of your ear is enough to uniquely identify you? and used to identify your relatives (it’s inherited).

To clarify: to obtain an E-passport you must submit a picture which is vouched for by an approved person – that’s scanned and the biometrics are encrypted into the passport after that data is entered into central passport database. As far as I know no one has managed to beat the E-passport encryption scheme – if you affix a false picture to the passport you won’t get past boarding.
I know of several cases where people used makeup to look like the photo on the passport and managed to board – but were arrested on embarking (hint: there are hight markers in the boarding corridor too, and not all camera view from above). Between ticketing and the plane your photo is taken – if you manage to avoid the camera you’ll be pulled out and checked on embarking – if you don’t and it’s not your passport you will be grabbed on embarking (like two case in China last year with people using latex masks). It’s not a perfect system – but it’s far harder to beat than just swapping the photo on the passport, which won’t get you on the plane – the passport is electronically checked to see if the photo biometrics matches the encrypted biometric embedded into the passport (not currently known to be forgable). There’s a reason why Irish passports are valued on the black market – mostly that they are easy to forge or modify.

Iain

herman • August 25, 2014 10:49 AM

There is an old saying that if you actually look like the picture on your passport, then you really need the vacation.

NobodySpecial • August 25, 2014 11:05 AM

vouched for by an approved person
That’s the weakness. You can have a completely forgery proof passport but the only proof of your identity is that your picture is signed by a respected member of the community – like a local religious leader.

dundee • August 25, 2014 11:06 AM

post mentions 15% false positive rate, but says nothing about false negatives..
I assume that false positives are cleared up after some time, but false negatives would be a bigger concern..

Daniel • August 25, 2014 11:43 AM

eh? Since when is 85% not very good? Any gambler would take 85% odd at the drop of a hat! I think this is one of those cases where people are letting the perfect be the enemy of the good. Let me ask this question: why do we need better than 85%? It’s not as if terrorist are sending 100 people through the border hoping that 15 make it through. No terrorist group could sustain an 85% loss rate, even in the short term.

Not only is 85% good, in most cases it’s all we need.

vas pup • August 25, 2014 11:52 AM

@ Andrew • August 25, 2014 7:35 AM
Your suggestion sounds reasonable and productive.
Second factor (not limited to that in your suggestion)used in the case of doubts based on the first factor usage only plus random selection of persons based on profiling (in good security sense of this word – not ‘political correctness bs’)

Anoni • August 25, 2014 12:46 PM

Maybe it’s time for two-factor identification on passports. When you register for your passport, you set a password/passphrase that is required when you present it at passport control.

How often do you use your passport? Most folks, who seldom travel out of the country, will just write the password down on a postit note and keep it in their passport. Otherwise they’ll never remember it.

Lane • August 25, 2014 1:10 PM

@Daniel
I will assume your comment about an 85% success rate as being acceptable was restricted to the specific domain of the topic under discussion which is photo matching.
An 85% success rate would be intolerable in almost every other important endeavor: Airline safety, drug manufacturing, food safety, electric power grids, workplace safety come to mind to name just a few.

Troels A. • August 25, 2014 1:44 PM

@Andrew: That solution would change nothing, and cause more problems than it causes. It would cause massive problems for tourism (if people forget their passwords for internet sites, they will also forget it for their passport, especially if they don’t travel often. And if you’re traveling as a family of, say, 5 people, you only need one forgotten password to have that vacation ruined). It would need to be implemented universally for it to be effective. And honestly, i don’t think it would be that effective in reality.

Jon • August 25, 2014 1:46 PM

Anybody who works with classifiers and detection can say that they’re giving half the number. They’re saying, for detecting passport fraud, the probability of missed detection is 0.15. Find, but what’s the probability of false alarm? Probably 0. The consequences of a false alarm (falsely detaining somebody) are probably so severe, that the thresholds are biased way towards missed detection.

Rich • August 25, 2014 1:54 PM

@ herman
I make it point to look distressed for official photos on the assumption that if I ever REALLY need to look like the picture that’s how I’ll be.

Mary • August 25, 2014 3:39 PM

I saw a documentary on one of the London airports. What they did when there was any question of a person at the airport being the person on the passport was to take a photograph of the person at the airport to compare with the passport photo. What they were dealing with was people claiming to be the person on the passport who were not. The problem of people not looking like their photos was mentioned. Also that when confronted with their photo alongside the passport, people admitted that the passport was not theirs.

So there is a solution, but it involves officials admitting that they are uncertain if the person and the passport match.

David Leppik • August 25, 2014 3:59 PM

The purpose of a passport or ID photo is, in my mind, to keep just anybody from being able to steal and use your passport/ID. A thief has to, at a minimum, steal from somebody who appears to be roughly the same age, race, gender, and build. If the person has glasses, the thief needs to get similar glasses. (Or steal them–and probably replace the lenses.)

Point being, the photo is doing its job if it discourages casual and even semi-serious thieves. The effort to steal a passport from an appropriate-looking person has to be more than the effort needed to fake a passport or replace the photo.

I’d guess that there are a lot of minorities who “all look the same” to most TSA agents, and the false negative rate goes way up for them. And I suspect Arabs are among them. Even so, the thief can’t rely on a particular TSA agent doing the check.

The fact that you don’t hear about people getting their passports stolen by doppelgangers suggests that the photo is doing its job.

DimmerThanThou • August 25, 2014 5:53 PM

Erm, maybe, as a newcomer, I’m missing something. But why is this even notable? Why would anyone expect otherwise? You’re all jumping to second or third order effects. Anyone heard of Darwin? ‘Selfish’ genetics, et cetera?

And what are the rates for different races? For observers from different social strata?

x • August 25, 2014 7:03 PM

East Germany (GDR) had a very simple but effective manual biometric approach: The boarder guards were trained in identifying ear shapes.

They had a list of numbered ear shapes (maybe 20) and would match those to the person. When you entered the GDR, you had to remove any headwear and turn your face so that they could see your ear.

Very low tech approach, but I always felt that they were much more effective than modern day boarder guards.
I heard somewhere that an East German stamp in a passport gave the passport authenticity even in the capitalist world, because the other boarder guards knew how strict the checks were over there.

Of course, their job was to keep their own citizens inside of the country, but they did it extremely well. Very few people managed to escape using fake West German passports, because they marked the passports in some way (ink or secret markings) when you entered the country. Many West Germans were willing to lend out passports (even some passport offices handed out blanks), but this route of escape only worked in the very early days. Despite a huge selection of pictures to choose from.

I’ve been searching without much success for more details about the technologies that they used. In general, they were all very low tech, but surprisingly hard to circumvent. For example, they would take a stamp pad and use several different inks on it. Each stamp impression thus got a unique color scheme. Then they would change the colors randomly throughout the day.

If anyone knows any details about what kind of secret inks and technologies are used, please post here. It is quite an interesting area where security by obscurity seems to be the guiding principle.

Phil • August 25, 2014 8:56 PM

I think this highlights the fact that relying on identity documents alone is unreliable. To me, this suggests that other detection methods (behavioural cues such as nervousness or hinkyness) should play a more important role.

Albert • August 26, 2014 5:34 AM

85% success rate not very good?

Jens Kubieziel • August 26, 2014 6:29 AM

I made a podcast with a researcher who works in the field of face recognition. He explained how people recognize faces and where we as people are good and where not. It seems that we can easily recognize people who we know (friends, colleagues etc.) and it is very hard to recognize unknown people. Small changes in exposure and even the same photo taken with different cameras leads to high error rates.

The podcast is available here: http://datenkanal.org/index.php?/archives/64-DK32-Gesichtserkennung.html (german only)

Chris Reynolds • August 26, 2014 9:34 PM

One of the reasons this came to light is there have been at least one, possibly two recent cases where people have left Australia using relatives passports, defying a travel ban. So two factor & even some biometrics people may not be very effective in these cases.

http://www.dailytelegraph.com.au/news/nsw/another-jihadist-passport-bungle-sydney-teen-leaves-country-using-brothers-passport/story-fni0cx12-1227024971748

As for the out of date passport photos, I assume that one is taken & kept each time you leave/enter a country, showing the gradual aging process?

Todd • August 27, 2014 12:49 PM

@dundee
The underlying paper says that 6% of valid photo-subject matches were wrongly rejected.

Also of interest, performance did not appear to be determined by either experience or by current training methods.

This reminds me of when I went through passport control going into Brezhnev’s Soviet Union. The passport officer was the youngest person in the airport. I felt sorry for him. It took a long time. He was more nervous than I was. I wonder if the punishment for a mistake makes any difference in performance.

RW Nye • August 28, 2014 3:05 AM

During the more than 25 years I worked for INS and CBP, I was routinely faced with the challenge of comparing people to their passport photos. Being fooled by a lookalike impostor was one of my constant concerns, and I would nearly always hold up the passport and compare the photo directly with the person standing in front of me. I must admit that fatigue after interacting with hundreds of air passengers in the course of a few hours may have had some impact on my accuracy, but I encountered relatively few suspected lookalikes. I don’t think I referred more than one or two individuals a month to secondary inspection for this particular reason. Photo-substituted documents, counterfeit visas and phony alien registration cards were much more common. And we did occasionally encounter passports that were counterfeit from cover to cover. The biggest threat, however, came from malafide applicants for admission whose paperwork was impeccable. They had fooled our overworked consular officials using fraudulent documentation of all types, and could be outed only through careful observation and questioning.

Evan • August 28, 2014 6:28 AM

@Scott:

As far as I know no one has managed to beat the E-passport encryption scheme – if you affix a false picture to the passport you won’t get past boarding.

Selection bias – if someone was able to fool that scheme, unless they were a security research wholly unaffiliated with government, of course you wouldn’t know about it, because there would be nothing to make anyone take notice.

TRX • August 30, 2014 9:35 PM

There are fewer unique faces than people.

I’m probably particularly poor at recognizing faces, given the number of times people have asked “do you remember who I am?”, but then I often encounter people who might as well be clones of people I know are thousands of miles away or deceased.

Can I pick a human face out of visual clutter? Yep, the human brain is hardwired for that. Can I pick out a particular human face out of a crowd? Not without spending considerable time examining every separate face… and if the sample size is larger enough, I’ll get multiple probables.

Gloria • January 25, 2015 6:15 PM

Maybe it’s because of plastic surgery. The people don’t get identified by their passport IDs because they did plastic surgery, and now the officer accuses them of stealing because they don’t look like before.

Schneier on Security

People Are Not Very Good at Matching Photographs to People

Comments

Leave a comment Cancel reply