More Research on the Effectiveness of Terrorist Profiling

Interesting:

The use of profiling by ethnicity or nationality to trigger secondary security screening is a controversial social and political issue. Overlooked is the question of whether such actuarial methods are in fact mathematically justified, even under the most idealized assumptions of completely accurate prior probabilities, and secondary screenings concentrated on the highest-probablity individuals. We show here that strong profiling (defined as screening at least in proportion to prior probability) is no more efficient than uniform random sampling of the entire population, because resources are wasted on the repeated screening of higher probability, but innocent, individuals. A mathematically optimal strategy would be ”square-root biased sampling,” the geometric mean between strong profiling and uniform sampling, with secondary screenings distributed broadly, although not uniformly, over the population. Square-root biased sampling is a general idea that can be applied whenever a ”bell-ringer” event must be found by sampling with replacement, but can be recognized (either with certainty, or with some probability) when seen.

Tags: , , ,

Posted on July 22, 2010 at 6:41 AM • 22 Comments

Comments

Imperfect Citizen • July 22, 2010 7:13 AM

Great article perhaps it speaks more to the targeting process than the actual on the ground process. I love this quote, can someone send it to the folks on the eastern seaboard?

“because resources are wasted on the repeated screening of higher probability, but innocent, individuals”

Sadly, there’s more damage than just resources here. For example, when these jobs on innocent people go on and on, like mine has for 3 years, the confidence of the observers is eroded in the agency ordering the observation. The damage to the observed person is awful, your entire sense of community is destroyed. Who do you go to for help when the FBI or NSA puts a terror label on you? Even when you are innocent people think there must be some reason the feds are watching you.

The article didn’t factor in fraud, and that contractors are doing the screening and when an innocent person is found to be innocent, even after 3 years of watching that means the gravy train is over for the local jd getting Patriot Act funds. Even the FBI and NSA aren’t looking at the possibility that federal contractors are not self policing.

In my case, when observers in larger cities are discussing me within my hearing “Is that her? Yes she’s waiting for her dad at the doctor’s” (I’m the only she in the waiting room) “I don’t know why we are still watching her, I think the hometown is the problem” That was in 9/08. Almost 2 years ago. Then he spoke of what was necessary to keep me “activated on the system” The system that makes me an “event”.
Apparently local jds have no supervision on Patriot Act observation jobs if other towns I travel to keep complaining. When I was in Kentucky the observers said “why did they pull us out of church for this? Can’t they see they are just a family?” Ironically, what the feds won’t confirm or deny is spoken of by observers. They aren’t cops, they aren’t trained in surveillance. They are people told that I am a terrorist.

The next question was “what are we going to do with her in church?” The what is form perimeters. That’s actually damaging to the observers, they have to watch me, change their seats, take photos, follow me in and out. Why? They are calling me an Arab terrorist. People point and say “does she look like a terrorist to you?”
I don’t speak the lingo, I’m not an Arab, I’m not a Muslim. Yet observers ask me to tell them about the Muslim religion, what country I’m really from, do I think we deserved the events of 9/11, etc. Its so frustrating. I called the FBI, the NSA, the DOJ IG. There’s no process to get help even after 3 count em 3 years of observation. I have a kid, he gets shunned and bullied because mom wears a terror label. They don’t stop watching me, my family, my friends, its awful.

Another point here is that while this is interesting, nobody factored in that we are in hard times. Observer jobs go the way of the 9-1-1 fellowship, friends and family of law enforcement, fire fighters, etc. Nobody factored in ignoring exculpatory information. With the Patriot Act there is no accounting, there is no scrutiny of the evidence. Nobody at the FBI or the NSA apparently asked why a woman under observation for over a year suddenly gets Arabic messages on her answering machine? Nobody asks how a woman who hangs around with nuns needs to be frisked by a man who then followed me on jobs after the frisking.

There is a lot of corruption where there is no transparency and no accountability. Normally its the FBI’s job to catch it. Apparently the contractors in this game are above the law. What makes it worse is that the contractors maintain/store the data used to keep innocent people on the job. Every day I hope that the DOJ IG will get permission to investigate Patriot Act Abuses. Nobody does. The agencies paying for these jobs don’t even care enough about the integrity of the data to investigate cases.

bill • July 22, 2010 7:59 AM

@ Imperfect Citizen
‘there is no transparency and no accountability’
— 2010AD

Quis custodiet ipsos custodes?
— 55AD

Ok, so why not make a start on transparency – Take photos of watchers, record conversations/interviews to mp3 player, then post them and any other activity to a blog.

If you can find a blog hosted in China that’ll be a hoot.

will • July 22, 2010 9:18 AM

<>

Hrm…

Who watches the watchmen.

…

While the threats are generally not so diverse internationally, there has to be some sort of profiling, unfortunately. It is important to do this without unfairly accusing innocent civilians, or souring relations with nations.

This is always the case in security, keeping out of false accusations yet pursuing theories until they are proven right or wrong.

Not difficult to do if one remains object and disapassionate, well coursed in the humanity of individuals.

So, in my opinion, what is most important is to simply treat every suspect with dignity and respect, “innocent until proven guilty”.

Some degree of pressure sometimes needs to be exerted – dispassionately, objectively – merely to see if there is any nervousness. To set people off guard.

That is a technique, however, and is limited… because anyone is going to be nervous and upset when pulled out of an airport.

And, all the problems arise from being discourteous to suspects, most of whom are innocent. The very vast majority.

Generally, setting them at ease, establishing rapport… is far more likely to get anything in that situation. Especially for would be terrorists indoctrinated in how “evil” we are.

They expect evil, not that, so that could throw them off right away, even systematically.

It is the same thing when someone normally rude or who could be rude is suddenly very nice… or where a situation you expected to be very unpleasant turns out to be otherwise.

The terrorist would invariably think the interviewers are “dumb” so would not worry about caution.

Even if this technique were made well known, it would still be effective.

My two cents.

kashmarek • July 22, 2010 9:24 AM

For the moment, I am going to equate the discussion to random selection (no pre-conditions, just pick somebody) and targeted selection (pre-conditions).

Assumming now, that due to prior failures, someone has made a case for a new initiative on terrorist profiling (random or targeted), with everything about the case being controversial, if approved it will either be successful or made to appear successful, simply to achieve the PR (public relations) requirements rather than safety (protection from terrorists).

Pretty sad.

Beta • July 22, 2010 9:48 AM

@will: “Some degree of pressure sometimes needs to be exerted – dispassionately, objectively – merely to see if there is any nervousness. To set people off guard.”

This takes my weekly prize for bad grammar and vague language used to mask a vile message.

Michael • July 22, 2010 10:21 AM

An interesting, but thoroughly useless result.

Interesting because my intuition would have been that if you knew the exact probability that any given traveler would accurately detected to be a terrorist by some test, then the optimal strategy for weighting travelers when selecting subjects for that test would be linear.

So if you could search either the Middle-Eastern-looking guy in his 20s (odds of finding him to be a terrorist, one in 10 million) or the Asian grandma (odds of finding her to be a terrorist, one in 10 billion), you should be 31 times more likely to search him and not her (not the 1000 times I would have expected).

Useless though, because we don’t have the inputs to the algorithm. I just made up those numbers in the previous paragraph from thin air, but it isn’t like TSA has better numbers.

That’s a correctable fault, at least hypothetically. In a more-perfect world, TSA could overcome its political and organizational problems and discover the actual odds (though if we are making up a more-perfect world, we should just make one up with fewer terrorists rather than better policemen).

What isn’t correctable is that actual terrorists prefer not to be caught. Any sort of static strategy will fail. A game-theoretic approach, one that realizes the opponent is intelligent and active, is what is called for.

BF Skinner • July 22, 2010 10:54 AM

@imperfect citizen ” jobs on innocent people go on and on, like mine has for 3 years”

What does your congressional rep have to say? Constitutent complaints, esp with the beauracracy, are their day job. Some have told me that theirs are very effective.

The lack of due process about the lists is a serious failing. I believe they are still stopping and searching a 4 year old boy every time he flies (marched with the real IRA he did). Any investigation that does not have a goal of either exoneration or a sanction is a fail.

We’ve discussed before the lack of discretion that TSA screeners have. They just plug a name into a pre-programmed resposne. LEOs have much greater lattitude. They are expected and encouraged to make deals with offendors so they can apphrend worse offendors.

I accept that information or behavior may require and trigger an investigation. People go off all the time and if there is evidence there should be investigation and resolution.

One investigtation I had was on a SysAdmin who may or may not have committed a felony to cover up a friend and co-worker’s misuse of computers and networks. We were certain of the facts. At a point where the co-worker may have become suspicious that their activity was being scruitized they asked for a re-image of their workstation. Not an unusual request but it could have been destruction of evidence and obstruction of justice. The SysAdmin performed the requested re-installation and we had to find out if they were complicit in an attempted coverup. After a lot of evidence collection and interviews we weighed the matter in a balance of evidence and could only fairly conclude – maybe. Maybe but, unproved and unprovable. At some point–the investigation has to concluded or dropped. And the intel world doesn’t work like that.

This doesn’t sound like what happened for imperfect citizen. It sounds like her investigators are stuck in the maybe loop. They’ve decided/been told what she is but haven’t/can’t collect evidence enough, but won’t close the file ’cause they know what she is GOTO.

Reminds me of the blood test device the police in Faboulous Furry Freak Brothers used to see if a hippy had drugs in them. The machine GOT it’s positive response even though it had to drain the hippie of every last drop his blood.

If the LEO don’t/can’t close the file at some point it becomes harrassment. (cf Kafka The Trial)

These lists aren’t allowing resolution and evidence to feedback into them. So you get 4 year old boys stuck on it. It took a major law in order for us to be able to correct the record that the credit agencies held on us. It will take legal action (and a lot of it) to do the same for the no-fly. Unfortunately not enough people are being effected on a routine basis.

There is this though and I don’t know imperfect citizen’s situtation but — if you aren’t a terrorist…

Are you something else?

This is a rhetorical question. Do not answer here. You do not have to defend yourself to us. But it’s something to consider.

There are all kinds of reasons (mostly unconstitutional but not all) why people with power may want you to know you are under scrutiny and to harrass you. There may be things in your life that are encouraging them a sideways approach that have nothing to do with terrorism.

Try your congressional reps, then see if ACLU or another advocate group can assist. @bills suggestion of lots of publicity may also be the only way out.

George • July 22, 2010 11:18 AM

By definition, someone who meets a profile is not “innocent.” They may not be a terrorist or guilty of any specific or relevant offense, but the mere fact that they fit a profile makes them inherently (and irrevocably) suspect.

If we’re going to rely on profiling as a vital “layer” of security (to use TSA terminology), we’re just going to have to accept that some (many?) innocent people will be burdened with having to prove their innocence every time they encounter a security checkpoint. This is the price we pay for the security benefit that profiling provides.

Of course, if the officials in charge of creating and implementing the profiles find that they’re wasting a lot of time and money on unproductive interrogations (i.e., false positives), they SHOULD feed that back to their bosses. The result should be either an evaluation and improvement of the profile, or abandoning the profiling if it proves ineffective.

Unfortunately, the “culture of security,” especially as practiced by the United States Department of Homeland Security’s bureaucracy, does not allow any sort of rational analysis. It’s all about posterior-covering and evading accountability for the inevitable failures. It’s more about the appearance of security than about actual security. Maintaining the illusion of infallibility is essential to that appearance.

Nobody wants to risk being blamed for “weakening security” by even suggesting the elimination of ineffective practices. So the costs and burdens continually pile up, perhaps aided by fudged data. They’ll report false positives as successes to create justification for burdensome and unpopular practices. Or they’ll make any relevant data classified, as much to protect themselves from embarrassing questions as to protect “national security.”

So while it’s certainly good to present objective facts and data about the effectiveness of the measures our Leaders are taking to keep us safe, it’s ultimately useless. That unaccountable bureaucracy is impervious to any of it, and may even retaliate against those who are persistent enough in raising questions and doubt to constitute a threat to the continued expansion of the bureaucracy.

BF Skinner • July 22, 2010 11:33 AM

@George “all about posterior-covering and evading accountability for the inevitable failures”

I find it good to avoid all or nothing constructions. They almost always wrong and if someone can find a single case that disproves your thesis then they can discount you — and do.

It’s why I like studies like this. It’s ‘common sense’ to look for Muslim extremists among Arab peoples. ‘Cause, says common sense only Arabs are Muslims. (there’s that all or nothing, two factor thinking again that’s gonna kill us)

Studies like this say “You may think that way person on the street. But facts don’t bear your thinking out.”

“I hate science,” says the person on the street, “I don’t understand it and it always tells me I’m wrong.” and goes to vote the Texas Board of Education life time tenure.

Fortunately policy wonks are more easily convinced by evidentiary fact based research than screeching angry people dressed like a trampy air line hostess.

MikeA • July 22, 2010 11:42 AM

“innocent unless proven guilty”. not “until“.
Big difference, and symptomatic of the problem.
When a person in power believes that it is not evidence, or lack thereof, that determines guilt, but merely persistence on his part, Bad Things happen.

George • July 22, 2010 11:54 AM

The bureaucratic “culture of security” also explains what “Imperfect Citizen” is suffering.

Assume for the moment that someone actually did investigate her and found a lack of evidence for any terrorist connection. (They probably didn’t do that, but let’s assume that they did.) Whoever did that investigation definitely does not want to risk being blamed in case she actually turned out to be a terrorist. After all, absence of evidence is not evidence of absence. And if there’s any doubt at all– which there clearly is, since someone had reason to put her on the list in the first place, even if he has no idea what that was– it’s always better to be safe than sorry when it comes to terrorism.

So she needs to remain on the list, rather than him or someone in his agency getting blamed in case something happens. Even if that something doesn’t involve “Imperfect Citizen,” he could be blamed for removing too many people from the list when they’re looking for scapegoats to protect the boss. He thus has nothing to gain from removing her from the list, and possibly much to lose!

Besides, the entire process is hidden behind a wall of secrecy. “Imperfect Citizen” has no way of even knowing who decided she is a terrorist or why. And she has no recourse or due process to do anything about it. I think we’d all agree that providing such a recourse would cause grievous harm to National Security, and would also divert resources from fighting terrorism. So nobody who puts people on lists has any incentive to remove anyone. If anything, they have every incentive not only to keep people on lists, but to add the name of anyone who might have any suspicious connections. It’s always better to be safe than sorry when it comes to terrorism!

For now there aren’t many people in “Imperfect Citizen’s” situation. The number is small enough that if the Homeland Security officials ever have to field questions about it they can convincingly argue that war inevitably causes injury to innocent bystanders. A thorough and effective net will inevitably catch a few suspicious people who are probably harmless, but we should regard that as evidence of the DHS’s diligence that keeps us safe. But as the number of “Imperfect Citizens” inevitably increases, they might not be able to get away with that.

frank • July 22, 2010 12:18 PM

@beta:”@will: “Some degree of pressure sometimes needs to be exerted – dispassionately, objectively – merely to see if there is any nervousness. To set people off guard.”

This takes my weekly prize for bad grammar and vague language used to mask a vile message.
”

Good example of doing exactly what he was saying while condemning it. Whether you did that with true conviction of condemnation or dispassionately I can not tell. I would guess the former.

I agree with your sentiment, that kind of trickiness is inhuman and not something we see in daily conversations ever. Only manipulative security people do that sort of thing.

And parents and kids.

Alex • July 22, 2010 12:41 PM

Oh no! Bruce, have you been reading posts by Eric S. Raymond again? You know you shouldn’t do that, you don’t know where he’s been!

🙂

moo • July 22, 2010 1:08 PM

@frank: Are you arguing that the message was not vile? Because it is. The U.S. is supposed to be the “Land of the Free” and the “Home of the Brave”. Everyone in it is supposed to be “innocent until proven guilty”. When you have security wonks following around a suspected terrorist for 3 years while she just goes about her day to day life, never doing anything to justify all the time and effort they are spending on their investigation, then it sounds like something is wrong. It sure sounds like the government is discriminating against her, and can’t or won’t make a clear case why. Like those people who are “randomly” singled out for SSSS treatment and extra interviews every time they fly, even though no one will tell them why. The answer is either because they fit a certain profile, or some unaccountable bureaucrat has added them to some secret list of “probably bad people” and there’s no way for them to find out what list they are on or get removed from it.

Do you think a society which harasses a subset of its members for years on end (even a small subset), when they don’t have enough evidence to charge and convict, is a just society? If you think this, you’re probably a white middle-class ordinary guy. Ask someone who’s been pulled over for “driving while black” and I bet they’ll have a different view from you.

I remember how shocked I was during a visit to Boston. A group of us were walking down the street on our way back to work, and we stopped at a convenience store while one of our group went in to get a drink. He was a black guy, and across the street was a white cop. The cop noticed my friend go into the convenience store, and he immediately crossed the street and followed him in. That’s racial profiling in action: Black guy going into a convenience store! Alert, alert, he’s probably going to rob the place or something! This asshole cop stood there in the store and watched him the entire time until he had paid for his drink and left. This guy had a bachelor’s degree in engineering and was working on his MBA, but to that cop he was a suspected criminal just because he walked into a convenience store while black.

Well that’s the same kind of treatment that Imperfect Citizen apparently gets: people watching her for no legitimate reason, all the time, in all kinds of public situations. The rest of us walk around in de-facto anonymity in public, we can go to the shopping mall and buy a new pair of shoes without being followed and peered at by secretive government types. Imagine how unnerving it must be to have your personal space invaded on a continual basis like that.

If freedom and liberty mean anything, they should mean that people are left alone to do what they will, unless there is a really good reason to hound after them. And if someone comes under a cloud of suspicion, they either need to prove it and charge them, or disprove it and then leave them alone. To allow it to go on for years, with no oversight or recourse, harms the liberty of all of us.

Bob_Dole • July 22, 2010 1:44 PM

It’s interesting to me that nobody here has noted that the first post seems like the product of textbook paranoid schizophrenia. So textbook, in fact, that I question whether it’s not cleverly tongue-in-cheek.

Zoolbia • July 22, 2010 6:47 PM

The problem with that analysis is the way it redefines the problem away from sampling a small population once, which is the actual problem faced by screeners in a given airport on a given day, to repeatedly sampling a large population, which is a “meta-problem” that doesn’t model the reality faced by screeners (though run in reverse it does model, to some extent, the reality faced by frequent travellers).

The authors explain first that sampling in order of priors is the best strategy for sampling a particular population once (which, to reiterate, closely models the actual physical problem). The authors then say that sampling with replacement in order of square-root of priors is the optimal strategy under a different model which doesn’t match reality so well.

So their work doesn’t point the way either they or Bruce seem to want it to: the paper actually says that airport screeners should sample according to priors!

There’s another problem. Secondary screening doesn’t detect terrorists, it detects weapons. The “priors” of interest point to terrorists, but screening passes over those terrorists who are merely travelling as opposed to those toting weapons for immediate “jihad operations.”

So the lack of “memory” in the screening process cuts two ways. It means that, viewed over time as from a great height, repeatedly sampling certain people wastes effort on those who will never meet the detection criterion (non-terrorists). On the other hand, repeated sampling is necessary to detect actual terrorists because we can’t always distinguish them (when they travel without weapons), so if our priors are valid, the fact that we end up repeatedly sampling likely terrorists is not wasteful but actually productive!

RH • July 22, 2010 6:49 PM

I think many are missing the point. The article is basically describing an upper bound on usefulness of profiling. Assuming your profile remains accurate, the sqrt() law described yields the upper bound on effectiveness.

Also important, it shows that it is possible to use profiling to improve the success rate of searches, given the arbitrarily difficult to achieve goal of a perfect profile.

The question of random sampling vs. profiling is one of metrics. If your metric is the average effort imposed on a citizen, then profiling is capable of yielding results on partial information. If your metric is the maximum effort imposed on a law abiding citizen (along the “driving while black” argument), then profiling will by definition fail, as the best thing you can do is distribute pain equally.

I believe the “right answer” lies between those two. “Maximum effort” is by definition the most extreme pareto optimality (everyone is the same). I believe society can gain nothing from a metric more extreme than “average effort,” though the ‘haves’ may disagree. Somewhere between those two is the “right” amount of profiling. I wish I knew where it was… I’d run for congress

averros • July 23, 2010 4:47 AM

“Who watches the watchmen.”

Well, well, well. From where I stand (a member of public) the three-letter bossy assholes do not look like watchmen. They look awfully like collaborators of the terrorists – actively furthering the declared goals of Mr. Osama ben Laden. You know, making US to go out of Middle East by destroying US economically and culturally.

That’s what terror is for, as a tactic. Causing the enemy to waste resources on the useless “security” and to cripple the civil society by sowing distrust and paranoia. The home-grown uniformed thugs are doing the ben Laden’s work.

Being a rational man, I worry a lot less about being killed by a terrorist bomb than about being mistakenly shot by one of the armed, aggressive, and terminally stupid members of the Reichssicherheitshauptamt , err, Homeland Security.

Imperfect Citizen • July 23, 2010 4:26 PM

Thanks for the remarks. I did try the whole route. Observers standing behind me in church said that I was put on a list for opposing torture by a Congressman and an FBI agent I had asked for help (just wrote letters no activism beyond that) letters that basically asked not to support x bill funding torture. Nuns wrote the letter “in the name of Jesus who we follow please do not fund x bill” Some of the funding was tied to School of the Americas. The ACLU website said people opposing torture such as School of Americas Watch were put on watch lists back in 07. That’s when my watch started.

Then I was studying at the library, I’m a stay at home mother, Catholic, not from the mid east, trying to work as a sub teacher so no criminal history. Seated near me are some 20 y/o people complaining about having to watch me. A man who had followed me before arrived and said “she named the person on the complaint the team will decide at the end of the week what to do” he pointed at me. I guess I named the person on my tapped phone as I was trying to discuss who did this to me in 2008. That was July 2008.
August 2008 I return from a two week vacation and there’s an Arabic sounding phone message. Sept 2008 other observers don’t know why I’m being watched.
I am working at getting mp3s. Its hard when you are a mom with a kid and you are hoping for a normal life. I’m not detective material. I went to my Senator and others. I can’t get one acknowledgment of my complaints to the DOJ IG.

Politically I’m not a radical. Everyone thinks its ridiculous that I’m being watched, even my observers.
I can only assume its fraud. One observer said my job is paid up to the end of the year.
I even contacted the NSA. Frankly, they were nice. Nicer than anyone. I figured the Arabic sounding phone message threw me in their pile. Plus observers were asking me what religion I really was and if I could tell them about Muslims.
My husband says its impossible to prove a negative. So its something the FBI can continue to fund until someone cuts off the cash.

If you are ever “watched” you will know it. My advice to you is don’t let on and do as others suggest, get mp3s. I can’t tell you how wearing it is never to be left alone. Even when observers are nice its awful. I appreciate your kind replies. It is odd to me that they keep it going each time I’ve been hurt or insulted I’ve met them with peaceful nonviolent responses. If I were the person they think I am, there would have been a sign of it by now.

I believe the people in our government are good, I believe in our system of government, I think that this system of watching is too new and fraud and abuse is too easy when public discussions are precluded by national security labels.

Thank you all for listening.

Dr. T • July 23, 2010 5:44 PM

The only way that random screening of an entire population can yield results that are no better than targeted screening based upon prior probabilities is if the ratio of the prior probabilities of the targeted population to the general population is essentially equal to 1.000. Therefore, the article has to be flawed. Such an outcome would be equivalent to claiming that random testing for HIV would yield equal results to targeted testing of people with higher prior probabilities of having HIV (such as intravenous drug users, people diagnosed with other sexually transmitted diseases, gays who engage in unprotected sex, people with unusual infections, etc.). Since we know that random screening for any disease is far less efficient than targeted screening, I cannot believe that random screening of all airline passengers will yield better anti-terrorism results than screening focused on young to middle-aged men who look like Arabs or who have passports from Afghanistan, Pakistan, Syria, Jordan, Egypt, Saudi Arabia, Iraq, Yemen, Iran, Lebanon, Libya, Chad, Sudan, Indonesia, etc. Anyone who claims otherwise doesn’t understand Bayesian statistics.

Of course, none of this gets to the real issue: the negative outcomes of screening passengers and preventing them from carrying weapons or objects that could be used as weapons. I would feel safer when flying if passengers were allowed to carry knives: a handful of terrorists would not be able to hijack a plane carrying dozens of knife-wielding passengers who know that a successful hijacking will result in their deaths.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.