Entries Tagged "profiling"

Page 2 of 6

Criminal Intent Prescreening and the Base Rate Fallacy

I’ve often written about the base rate fallacy and how it makes tests for rare events—like airplane terrorists—useless because the false positives vastly outnumber the real positives. This essay uses that argument to demonstrate why the TSA’s FAST program is useless:

First, predictive software of this kind is undermined by a simple statistical problem known as the false-positive paradox. Any system designed to spot terrorists before they commit an act of terrorism is, necessarily, looking for a needle in a haystack. As the adage would suggest, it turns out that this is an incredibly difficult thing to do. Here is why: let’s assume for a moment that 1 in 1,000,000 people is a terrorist about to commit a crime. Terrorists are actually probably much much more rare, or we would have a whole lot more acts of terrorism, given the daily throughput of the global transportation system. Now lets imagine the FAST algorithm correctly classifies 99.99 percent of observations—an incredibly high rate of accuracy for any big data-based predictive model. Even with this unbelievable level of accuracy, the system would still falsely accuse 99 people of being terrorists for every one terrorist it finds. Given that none of these people would have actually committed a terrorist act yet distinguishing the innocent false positives from the guilty might be a non-trivial, and invasive task.

Of course FAST has nowhere near a 99.99 percent accuracy rate. I imagine much of the work being done here is classified, but a writeup in Nature reported that the first round of field tests had a 70 percent accuracy rate. From the available material it is difficult to determine exactly what this number means. There are a couple of ways to interpret this, since both the write-up and the DHS documentation (all pdfs) are unclear. This might mean that the current iteration of FAST correctly classifies 70 percent of people it observes—which would produce false positives at an abysmal rate, given the rarity of terrorists in the population. The other way of interpreting this reported result is that FAST will call a terrorist a terrorist 70 percent of the time. This second option tells us nothing about the rate of false positives, but it would likely be quite high. In either case, it is likely that the false-positive paradox would be in full force for FAST, ensuring that any real terrorists identified are lost in a sea of falsely accused innocents.

It’s that final sentence in the first quoted paragraph that really points to how bad this idea is. If FAST determines you are guilty of a crime you have not yet committed, how do you exonerate yourself?

Posted on May 3, 2012 at 6:22 AM • View Comments

Profiling Lone Terrorists

Masters Thesis from the Naval Postgraduate School: “Patterns of Radicalization: Identifying the Markers and Warning Signs of Domestic Lone Wolf Terrorists in Our Midst.”

Abstract:

This thesis will scrutinize the histories of our nation’s three most prolific domestic lone wolf terrorists: Tim McVeigh, Ted Kaczynski, and Eric Rudolph. It will establish a chronological pattern to their radicalization and reveal that their communal ideological beliefs, psychology, attributes, traits, and training take place along a common chronological timeline. Their pattern of radicalization can be used as an indicator of lone wolf terrorist radicalization development in future cases. This thesis establishes a strikingly similar chronological pattern of radicalization that was present in each terrorist’s biography. This pattern can identify future lone wolf terrorist radicalization activity upstream. It can provide a valuable portent to apply in the analysis of potential lone terrorists, potentially enabling law enforcement to prevent tragedies emerging from the identified population through psychological assistance, evaluation, training, or, in the worst case, detention.

Paper.

Posted on December 7, 2010 at 6:43 AM • View Comments

More Research on the Effectiveness of Terrorist Profiling

Interesting:

The use of profiling by ethnicity or nationality to trigger secondary security screening is a controversial social and political issue. Overlooked is the question of whether such actuarial methods are in fact mathematically justified, even under the most idealized assumptions of completely accurate prior probabilities, and secondary screenings concentrated on the highest-probablity individuals. We show here that strong profiling (defined as screening at least in proportion to prior probability) is no more efficient than uniform random sampling of the entire population, because resources are wasted on the repeated screening of higher probability, but innocent, individuals. A mathematically optimal strategy would be ”square-root biased sampling,” the geometric mean between strong profiling and uniform sampling, with secondary screenings distributed broadly, although not uniformly, over the population. Square-root biased sampling is a general idea that can be applied whenever a ”bell-ringer” event must be found by sampling with replacement, but can be recognized (either with certainty, or with some probability) when seen.

Posted on July 22, 2010 at 6:41 AM • View Comments

Racial Profiling No Better than Random Screening

Not that this is any news, but there’s some new research to back it up:

The study was performed by William Press, who does bioinformatics research at the University of Texas, Austin, with a joint appointment at Los Alamos National Labs. His background in statistics is apparent in his ability to handle various mathematical formulae with aplomb, but he’s apparently used to explaining his work to biologists, since the descriptions that surround those formulae make the general outlines of the paper fairly accessible.

Press starts by examining what could be viewed as an idealized situation, at least from the screening perspective: a single perpetrator living under an authoritarian government that has perfect records on its citizens. Applying a profile to those records should allow the government to rank those citizens in order of risk, and it can screen them one-by-one until it identifies the actual perpetrator. Those circumstances lead to a pretty rapid screening process, and they can be generalized out to a situation where there are multiple likely perpetrators.

Things go rapidly sour for this system, however, as soon as you have an imperfect profile. In that case, which is more likely to reflect reality, there’s a finite chance that the screening process misses a likely security risk. Since it works its way through the list of individuals iteratively, it never goes back to rescreen someone that’s made it through the first pass. The impact of this flaw grows rapidly as the ability to accurately match the profile to the data available on an individual gets worse. Since we’ve already said that making a profile is challenging, and we know that even authoritarian governments don’t have perfect information on their citizens, this system is probably worse than random screening in the real world.

In the real world, of course, most of us aren’t going through security checks run by authoritarian governments. In Press’ phrasing, democracies resample with replacement, in that they don’t keep records of who goes through careful security screening at places like airports, so people get placed back on the list to go through the screening process again. One consequence of this is that, since screening resources are never infinite, we can only resample a small subset of the total population at any given moment.

Press then examines the effect of what he terms a strong profiling strategy, one in which a limited set of screening resources is deployed solely based the risk probabilities identified through profiling. It turns out that this also works poorly as the population size goes up. “The reason that this strong profiling strategy is inefficient,” Press writes, “is that, on average, it keeps retesting the same innocent individuals who happen to have large pj [risk profile match] values.”

According to Press, the solution is something that’s widely recognized by the statistics community: identify individuals for robust screening based on the square root of their risk value. That gives the profile some weight, but distributes the screening much more broadly through the population, and uses limited resources more effectively. It’s so widely used in mathematical circles that Press concludes his paper by writing, “It seems peculiar that the method is not better known.”

Other articles on the research here, here, and here. Me on profiling.

Posted on February 4, 2009 at 12:50 PM • View Comments

MI5 on Terrorist Profiling

There’s no profile:

MI5 has concluded that there is no easy way to identify those who become involved in terrorism in Britain, according to a classified internal research document on radicalisation seen by the Guardian.

[…]

The main findings include:

• The majority are British nationals and the remainder, with a few exceptions, are here legally. Around half were born in the UK, with others migrating here later in life. Some of these fled traumatic experiences and oppressive regimes and claimed UK asylum, but more came to Britain to study or for family or economic reasons and became radicalised many years after arriving.

• Far from being religious zealots, a large number of those involved in terrorism do not practise their faith regularly. Many lack religious literacy and could actually be regarded as religious novices. Very few have been brought up in strongly religious households, and there is a higher than average proportion of converts. Some are involved in drug-taking, drinking alcohol and visiting prostitutes. MI5 says there is evidence that a well-established religious identity actually protects against violent radicalisation.

• The “mad and bad” theory to explain why people turn to terrorism does not stand up, with no more evidence of mental illness or pathological personality traits found among British terrorists than is found in the general population.

• British-based terrorists are as ethnically diverse as the UK Muslim population, with individuals from Pakistani, Middle Eastern and Caucasian backgrounds. MI5 says assumptions cannot be made about suspects based on skin colour, ethnic heritage or nationality.

• Most UK terrorists are male, but women also play an important role. Sometimes they are aware of their husbands’, brothers’ or sons’ activities, but do not object or try to stop them.

• While the majority are in their early to mid-20s when they become radicalised, a small but not insignificant minority first become involved in violent extremism at over the age of 30.

• Far from being lone individuals with no ties, the majority of those over 30 have steady relationships, and most have children. MI5 says this challenges the idea that terrorists are young men driven by sexual frustration and lured to “martyrdom” by the promise of beautiful virgins waiting for them in paradise. It is wrong to assume that someone with a wife and children is less likely to commit acts of terrorism.

• Those involved in British terrorism are not unintelligent or gullible, and nor are they more likely to be well-educated; their educational achievement ranges from total lack of qualifications to degree-level education. However, they are almost all employed in low-grade jobs.

Posted on August 22, 2008 at 6:18 AM • View Comments

Automatic Profiling Is Useless

No surprise:

Automated passenger profiling is rubbish, the Home Office has conceded in an amusing—and we presume inadvertent—blurt. “Attempts at automated profiling have been used in trial operations [at UK ports of entry] and has proved [sic] that the systems and technology available are of limited use,” says home secretary Jacqui Smith in her response to Lord Carlile’s latest terror legislation review.

The U.S. wants to do it anyway:

The Justice Department is considering letting the FBI investigate Americans without any evidence of wrongdoing, relying instead on a terrorist profile that could single out Muslims, Arabs or other racial or ethnic groups.

I’ve written about profiling before.

Posted on July 7, 2008 at 1:37 PM • View Comments

1 2 3 4 … 6

Sidebar photo of Bruce Schneier by Joe MacInnis.