Pentagon May Issue Pocket Lie Detectors to Afghan Soldiers

This is just ridiculous. Lie detectors are pseudo-science at best, and even the Pentagon knows it:

The Pentagon, in a PowerPoint presentation released to msnbc.com through a Freedom of Information Act request, says the PCASS is 82 to 90 percent accurate. Those are the only accuracy numbers that were sent up the chain of command at the Pentagon before the device was approved.

But Pentagon studies obtained by msnbc.com show a more complicated picture: In calculating its accuracy, the scientists conducting the tests discarded the yellow screens, or inconclusive readings.

That practice was criticized in the 2003 National Academy study, which said the “inconclusives” have to be included to measure accuracy. If you take into account the yellow screens, the PCASS accuracy rate in the three Pentagon-funded tests drops to the level of 63 to 79 percent.

Tags: , , , ,

Posted on April 14, 2008 at 12:57 PM • 45 Comments

Comments

Andre LePlume • April 14, 2008 1:28 PM

I’m sure the experimental conditions under which the PC-ASS were tested (US soldiers, in the US, not suspected of wrongdoing) are identical to those in the field. Consequently, biological indicators of nervousness are equally unlikely to be present. After all, why would a possible “foreign fighter”, speaking english as his second language, be at all nervous when using the PC-ASS in a war zone, under the direction of an armed infantryman?

Aaron • April 14, 2008 1:30 PM

Isn’t the principal virtue of lie detectors that the person being questioned is worried that they work?

AngryAfgan • April 14, 2008 1:37 PM

@stephen, what nonsense, do you seriously think you should you convict someone because on the average the detector is 63-79% accurate?

Stats • April 14, 2008 1:50 PM

Funny numbers, there.
“For every 100 deceptive people, the researchers reported, the device would detect 86 (red), with two false negatives (green) and 12 uncertain (yellow).

For every 100 truthful people, they said, it would detect 50 (green), with eight false positives (red) and 42 uncertain (yellow).”

I read this as stating that in a population consisting of 2/3 truthful answers and 1/3 lies, the results are split evenly between red, yellow and green. Well, 34:32:34 but I call that even. After tossing the yellows out of the consideration one might as well be using a PRNG to get the results…

What for? • April 14, 2008 1:55 PM

@AngryAfgan

“do you seriously think you should you convict someone because on the average the detector is 63-79% accurate?”

Who said anyone is getting “convicted”?

Are the Afgan soldiers using these detectors in police stations?

Or on the street? They’re ‘pocket’ lie detectors.

Carlo Graziani • April 14, 2008 2:07 PM

While I’m pleased that that NAS polygraph study is at least being cited, it would have been nice if the reporter had actually bothered to read it.

The study found that to the extent that polygraphing is of any value at all, it is in the context of an investigation. That is, you have a suspect of a crime, and you ask them questions about that crime. Those circumstances change the system response curve, so that there is arguably a “sweet spot” of sensitivity threshold where a useful true positive rate is attended by an acceptably low false positive rate.

For screening — no suspect, no crime, just trawling for interesting physio response — the study found that polygraphing is totally useless. The response curve has no sweet spot — depending on the sensitivity threshold you either detect no security risks or you accept a flood of false positives. As a screening tools, polygraphs are about as reliable as ouija boards.

This is the reason DOE deep-sixed the report as soon as it was prepared (at Congress’ request). The Federal Securocracy has a near religious faith in polygraph screening, despite the fact that the spies we actually do catch from time to time all consistently pass their polygraph screening tests.

The distinction between investigation and screening is an important one, but it is rarely made, because it is useful to polygraph advocates to conflate “accuracy” calibration numbers for the two activities — investigation accuracy is used to promote the totally inaccurate screening use, as with the Afghan PCASS deployment. Sadly, the media is still awfully easy to snow with this kind of switch.

Random Guy • April 14, 2008 2:21 PM

No, of course they wouldn’t convict someone based on these results, but we’re not trying to do that. We’re just deciding whether to shoot someone.

Anonymous • April 14, 2008 2:30 PM

Stephen: That depends. If you test your gadget on a population of people who tell the truth three times out of four, even a trivial “lie detector” that always answers TRUTH will get a 75% success rate.

Percentage reports that are not accompanied by a discussion of the relation between false positives and false negatives are inherently suspicious. If tests are done with a 50-50 distribution of lying and hones subjects, the compound success percentage tells very little about performance in the field (where the majority of interviewees are non-terrorists), and if the tests attempt to recreate a anything like a realistic distribution, the device can reach optimal reliability simply by hard-wiring a “green” answer.

Jason • April 14, 2008 2:34 PM

quote from article
“Let’s take a worst-case scenario here, and let’s say PCASS really is 60 percent accurate,” said Krapohl, who heads the project for the Defense Academy for Credibility Assessment at Fort Jackson, S.C. “So let’s get rid of the PCASS because it makes errors, and go back to the approach we’re currently using, which has less accuracy? As you can see, that’s really quite untenable if we’re interested in saving American lives and serving the interests of our commanders overseas.”

= = = =

W. T. F.

Total reality disconnect.

We will have untrained (or minimally trained) soldiers relying on a quacky piece of hardware.

The results they obtain will give them false confidence and the fact that they rely on them will spread fear among the innocent.

Stats • April 14, 2008 2:46 PM

@Anonymous at April 14, 2008 02:30 PM
Exactly.
The device may well appear to be excellent at detecting lies as long as most answers are lies.
If, on average, only one out of every 12 answers is a lie, the interpretation of the results is like this instead:
Green: truth (~99,5% confidence)
Yellow: truth (~97,5% confidence)
Red: truth or lie (~50,5%/~49,5%) or toss a fair coin.

David Harmon • April 14, 2008 2:50 PM

So let’s see:

It’s only for use on the natives, not on U.S. Personnel… and note the repeated concern for “protecting” the troops, but no mention that falsely accusing locals of this week’s incident, might itself cause some trouble.

They were calibrated and tested on American soldiers, but they’re intended for use on Afghan locals. Between cultural,racial, and training differences, could there possibly be some differences in psychological responses or bodily reactions between those two groups? I guess we’ll never know, since “It’s virtually impossible to do a validity study in a war zone,” Krapohl said.

Using the numbers copied by “Stats”, if used on a group of which 10% are lying, it will accuse almost as many innocents as liars (7.2 to 8.6), and if 90% are lying, it will clear only half the truth-tellers.

There’s no mention of the device making any record of its reading, much less the data leading to it. Accountability, much?

The article itself includes some outrageous statements: Krapohl acknowledges that possibility, saying it’s not dissimilar from the ways colleges choose which students to admit. Oh, aside from the guys with loaded guns, that is!

This is going to be a disaster. Indeed, that guy’s name is quite appropriate to his project….

Anonymous • April 14, 2008 2:57 PM

Bah, no italics? Sorry, my penultimate paragraph should have read:

The article itself includes some outrageous statements: “Krapohl acknowledges that possibility, saying it’s not dissimilar from the ways colleges choose which students to admit.” Right, aside from the guys with loaded guns, that is!

Harry • April 14, 2008 2:59 PM

Carlo Graziani is correct.

Another way to think about the issue is with an analogy to IQ tests. The standard IQ test was designed to be given to small children, by trained testers, in a one-on-one environment, and to yield a relative measure of that child’s insights as compared to other children of the same age. (One implication is that one’s IQ changes over time.) When the test was administred under the designed circumstances, it worked pretty well. But when an adult does an online standardized IQ quest the results are worthless.

Polygraphs are similar. They are designed to be adminstered by trained personnel, who know about the subject and his background, who know why the subject is being questioned (ie, about the case), and who know what topics the investigators want to know about. This prep takes time. The test is administered one-on-one, is done in combination with a conversation with the subject, and there’s always the possibility of needing another testing to clarify.

Now tell me, class, how many of these circumstances will be reproduced by the pocket lie detector, used in the street, by soldiers, on random folk passing by?

markm • April 14, 2008 3:09 PM

Even a device that gets it right most of the time for both truth-tellers and liars can reduce the effectiveness of investigations or screening if the user comes to depend on the device and stops cross-checking stories and otherwise using his own mind.

Anonymous • April 14, 2008 3:10 PM

Might as well as join the Church of Scientology and use one of their E-Machines on captives. Mind control at its best.

LC • April 14, 2008 3:14 PM

Why do they need pocket little lie detector machines. They have guns and they are “in control” so they can just beat the crap out of those people until they tell them the real truth: which is that they are terrorists planning to bomb stuff.

The US has to follow laws, but they get to play dirty, its just not fair.

@tangerine blue: no need to thank me for my enlightening commentary.

Anon Cowerd • April 14, 2008 3:17 PM

@harry
do you meen that the online iq test i took and scored 152 is not wurf anething?

PKS • April 14, 2008 4:15 PM

There was a parliamentary inquiry back in the 80’s, on the use of the polygraph in Canada.

Amongst other things, they found that you could probably replace the polygraph with a purely theatrical device (a topic explored for comic relief on HBO’s “the Wire”), and get about the same results as the polygraph.

This is the same old ‘false positive’ problem. If there’s an order of magnitude difference between your accuracy and the actual incidence rate, it’s worse than useless.

Consider a hypothetical case of 10000 people, 100 of whom are terrorists, and trying to screen them with a lie detector that is 80% accurate. You’ll get:
-80 actual terrorists caught;
-20 actual terrorists pass the test, they get jobs in the army or police force;
-2000 people falsely accused of terrorism, locked up, maybe beaten or tortured as well.

If your accuracy rate is 63%, you can expect to have 3700 innocent people in jail.

If this is “better than what they’re doing now”, then what pray tell are they doing now?

Roy • April 14, 2008 4:43 PM

Given that the sensitivity is .86 and the specificity is .50, then we would expect the gadget to be right virtually all the time when every subject tells the truth every time, or when every subject lies every time. When there are as many lies as truths, the gadget would be right about 71% of the time.

A second problem is that the laboratory testing is artificial. The truth is always known, and the subjects always know the experimenters already know the truth or will soon enough know the truth. In the field, the truth is not known in the majority of cases, so there is no basis for evaluation.

BTW, accuracy of detection is function of sensitivity, specificity, and prevalence. Whenever ‘accuracy’ is claimed without revealing the three factors determining it, be very very skeptical.

Orv • April 14, 2008 6:22 PM

I don’t know about pocket devices, but polygraphs themselves are total juju. The way it works, is the system is calibrated by the tester catching you in something that they are pretty sure (guess) is a lie around something that they (the human tester, say 40 hours of training and no post secondary degree) try and catch you on, say drinking and driving or premarital/adulterous sex or casual drug use in your youth. This is the baseline used thereafter.

The rare totally honest captain truheart type guy will pretty much always fail a polygraph. To pass a polygraph, you actually have to lie, as the system is based on the (probably not totally inaccurate assumption) that all human beings are inherently liars, and lie about the things that the tester thinks they will lie about. Which assumption says something about police & security type mind-sets, at a minimum.

Terry Cloth • April 14, 2008 8:47 PM

From the article: “Only ‘a triage device’
Pentagon officials say that the new device will not be used to make final decisions, that the rules forbid it.”

You’re out in Iraq, some of the people want to shoot, but you don’t know which.
You want every edge you can get.
You’ve got this lie detector.

What’s the chance that the “preliminary” aspect of it is going to survive a week?
Whatever the Army does, its output it going to be taken as gospel real fast. These folks are not in a position to take their time to make nice distinctions.

“It’s red? Great, we caught one!”

Terry Cloth • April 14, 2008 8:53 PM

Typo! The first sentence of my previous comment should read:

You’re out in Iraq, some of the people want to shoot you, but you don’t know which.

Ctrl-Alt_Del • April 15, 2008 12:29 AM

2.4 The PCASS shall not be used to test U.S. persons.

4.3 The PCASS operator must obtain the examinees voluntary consent before conducting an examination.

4.5 PCASS examinations files shall be maintained for at least 20 years.

Hmm.

US citizens can’t be tested because … ?

The test result must be kept for at least 20 years. Who wants to live 20 years with such a Sword of Damocles hanging over their head? Even if the device is only used for screening now, such records could be mangled and misused a hundred ways in 20 years, mostly to the suspect’s detriment – even if they are totally innocent now. And what happens if the suspect, as is only sensible, refuses to be tested by any such shonky device?

Ctrl-Alt-Del • April 15, 2008 1:20 AM

Actually I can answer my own last question.

“what happens if the suspect, as is only sensible, refuses to be tested by any such shonky device?”

The obvious answer is, they won’t pass “GO” and won’t get the job.

Ctrl-Alt-Del • April 15, 2008 1:24 AM

… And when they try to pass through a military checkpoint on their way home that night, their name will have been flagged in a military database and they will be held for further questioning.

Sejanus • April 15, 2008 2:57 AM

50/50 is purely random method. Any intelligent person would do better than that. I’d say, 60 – 70%.

D0R • April 15, 2008 4:21 AM

US citizens can’t be tested because … ?

Because the test involves getting rid of fair judgment, presumption of innocence, and other ridiculously small details concerning human rights that however cannot be waived so easily when testing an American person.

bob • April 15, 2008 6:58 AM

This is not about using the device to determine who you are going to shoot. It is worse than that – it is being used to determine who you are going to trust to stand behind you with a loaded firearm.

I suspect a trained interrogator would have a better accuracy than this overpriced parlor game.

Somebody who makes these toys must have a brother in procurement at the pentagon.

Reminds me of some (very expensive) “video conference” devices they issued us to take to the field. The feed was not compressed in any way; if you made the screen as small as possible and cut the frame rate as low as possible it could be cut down from 6MB/s to 2.2MB/s. On a base that had a 1.5MB/s pipe to support all the info services on the entire base (secure network, unsecure network, phones etc). In other words the farthest we could videoconference would be the other side of our own base, not back to the US.

Eponymous • April 15, 2008 7:53 AM

I would hope they would not indict or evaluate people strictly on the lie detection, but in an environment where people are unused to technology, cultivating belief in the lie detector may help it to be more useful in getting actual first hand confessions.

Mack • April 15, 2008 8:17 AM

And, what pray tell is the cost of these little wonders?
A hallmark of the current struggles is that we pay top dollar to private industry for ‘solutions’ which make us less safe.

LC • April 15, 2008 8:38 AM

@Mack: Trickle down economics!! DUH!! we need something to kick this economy in the but, and by making the rich richer, we are benefiting the whole of America.

It all trickles down.

gfujimori • April 15, 2008 9:42 AM

One of the questions none of these studies has explored is: what’s the success rate with people who know that polygraph devices are complete quackery?

On another note, I like thinking about 50 years from now and wondering how scientists then will consider the ruling elite of today in most Western countries. If my guess is right – they’ll pretty much be viewed as medieval primates considering the way they embrace nonsense like this.

shadow • April 15, 2008 9:49 AM

not as much “juju” as you might think… although a soldier would probably not be the best person to administer a polygraph.

this is a nice overview of older soviet polygraph information

http://www.foia.cia.gov/search.asp?pageNumber=2&freqReqRecord=undefined&refinedText=undefined&freqSearchText=undefined&txtSearch=undefined&exactPhrase=&allWords=EEG+soviet&anyWords=&withoutWords=&documentNumber=&startCreatedMonth=1&startCreatedDay=1&startCreatedYear=1940&endCreatedMonth=4&endCreatedDay=15&endCreatedYear=2008&startReleasedMonth=1&startReleasedDay=1&startReleasedYear=1940&endReleasedMonth=4&endReleasedDay=15&endReleasedYear=2008&sortOrder=DESC#

rai • April 15, 2008 9:52 AM

Will bush be using this next time he makes a significant speach with the red green readout scrolling along the bottom of the screen? Oh I guess not, its only to sort out who gets “cheney’d” (we don’t torture, we cheney you to the ceiling in a dark room for a few weeks.

actually there was a voice stress anylizer that was used on Sadats address to the Knesset back in the day.

the idea of a screen crawl of a vsi during such crimes as the runup to bushs hobby war would have been a good use to put it to. Of course, my own bushit detector has been working fine since the coup of 2000. Lets stop being wrong about bulls and put it on the bush where it belongs, Bushit.

Doug in Seattle • April 15, 2008 11:24 AM

Bruce,

How much of the knee-jerk security proposals are what I like to call “we must do something”? I have had this discussion with my wife, who tends to be much more fearful of the wildly improbable. Whenever I remark on how airport security is a sham, she says “Well, we must do something.” I like to point out that doing the wrong thing, even if for the right reasons, is worse than doing nothing at all.

doug

David Harmon • April 15, 2008 4:48 PM

LC: “It all trickles down”

That don’t smell like rain to me! 😉

Doug: As Bruce has put it before, “Something must be done, this is something, therefore we will do it!”. Even so, my bet is, this came out of a pork barrel gone rancid.

Peter E Retep • April 15, 2008 5:31 PM

“63-79% accurate?” Without the uncertain 33%.
Does this mean that 70% give the right (correct) answers to the right (correct) questions?

Deconstruction requires Reconstruction for interpretation.

Might this rather mean that of the 1-in-20 persons selected correctly to ask questions of – who are otherwise invisible among the other 20 suspected of guilty knowledge,

that the questioners can be sure that the answers are believed by the answerer only 2/3’s of the time, unless the answerer is too annoyed, alarmed, or agitated
by the question, the questioner, or the circumstances presented or raised.

Since the anticipated responses to this circumstance are violence, torture, death, imprisonment, or arrest,
the whole interaction is weighted and biased toward indicating a stressed reaction to the question –
(“Aw, this is getting too complicated. Just terminate.”)

paul • April 15, 2008 8:36 PM

What do you want to be that in a war zone, with untrained soldiers using this thing, every “yellow” reading is going to be treated exactly the same as “red”? I know that if I were stuck in the middle of hostile territory and my gizmo told me a suspect might be lying (and it certainly couldn’t promise they were telling the truth) I’d want to act as if I’d just met a hostile.

Of course, that usage would mean treating 58% of truthful respondents as hostile, but what the heck.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.