Schneier on Security
A blog covering security and security technology.
« Michael Chertoff on Google Glass |
| Is the U.S. Government Recording and Saving All Domestic Telephone Calls? »
May 7, 2013
Intelligence Analysis and the Connect-the-Dots Metaphor
The FBI and the CIA are being criticized for not keeping better track of Tamerlan Tsarnaev in the months before the Boston Marathon bombings. How could they have ignored such a dangerous person? How do we reform the intelligence community to ensure this kind of failure doesn't happen again?
It's an old song by now, one we heard after the 9/11 attacks in 2001 and after the Underwear Bomber's failed attack in 2009. The problem is that connecting the dots is a bad metaphor, and focusing on it makes us more likely to implement useless reforms.
Connecting the dots in a coloring book is easy and fun. They're right there on the page, and they're all numbered. All you have to do is move your pencil from one dot to the next, and when you're done, you've drawn a sailboat. Or a tiger. It's so simple that 5-year-olds can do it.
But in real life, the dots can only be numbered after the fact. With the benefit of hindsight, it's easy to draw lines from a Russian request for information to a foreign visit to some other piece of information that might have been collected.
In hindsight, we know who the bad guys are. Before the fact, there are an enormous number of potential bad guys.
How many? We don't know. But we know that the no-fly list had 21,000 people on it last year. The Terrorist Identities Datamart Environment, also known as the watch list, has 700,000 names on it.
We have no idea how many potential "dots" the FBI, CIA, NSA and other agencies collect, but it's easily in the millions. It's easy to work backwards through the data and see all the obvious warning signs. But before a terrorist attack, when there are millions of dots -- some important but the vast majority unimportant -- uncovering plots is a lot harder.
Rather than thinking of intelligence as a simple connect-the-dots picture, think of it as a million unnumbered pictures superimposed on top of each other. Or a random-dot stereogram. Is it a sailboat, a puppy, two guys with pressure-cooker bombs, or just an unintelligible mess of dots? You try to figure it out.
It's not a matter of not enough data, either.
Piling more data onto the mix makes it harder, not easier. The best way to think of it is a needle-in-a-haystack problem; the last thing you want to do is increase the amount of hay you have to search through. The television show Person of Interest is fiction, not fact.
There's a name for this sort of logical fallacy: hindsight bias. First explained by psychologists Daniel Kahneman and Amos Tversky, it's surprisingly common. Since what actually happened is so obvious once it happens, we overestimate how obvious it was before it happened.
We actually misremember what we once thought, believing that we knew all along that what happened would happen. It's a surprisingly strong tendency, one that has been observed in countless laboratory experiments and real-world examples of behavior. And it's what all the post-Boston-Marathon bombing dot-connectors are doing.
Before we start blaming agencies for failing to stop the Boston bombers, and before we push "intelligence reforms" that will shred civil liberties without making us any safer, we need to stop seeing the past as a bunch of obvious dots that need connecting.
Kahneman, a Nobel prize winner, wisely noted: "Actions that seemed prudent in foresight can look irresponsibly negligent in hindsight." Kahneman calls it "the illusion of understanding," explaining that the past is only so understandable because we have cast it as simple inevitable stories and leave out the rest.
Nassim Taleb, an expert on risk engineering, calls this tendency the "narrative fallacy." We humans are natural storytellers, and the world of stories is much more tidy, predictable and coherent than the real world.
Millions of people behave strangely enough to warrant the FBI's notice, and almost all of them are harmless. It is simply not possible to find every plot beforehand, especially when the perpetrators act alone and on impulse.
We have to accept that there always will be a risk of terrorism, and that when the occasional plot succeeds, it's not necessarily because our law enforcement systems have failed.
This essay previously appeared on CNN.
EDITED TO ADD (5/7): The hindsight bias was actually first discovered by Baruch Fischhoff: "Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty," Journal of Experimental Psychology: Human Perception and Performance, 1(3), 1975, pp. 288-299.
Posted on May 7, 2013 at 6:10 AM
• 39 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
I'll ask a question I've asked elsewhere here - hopefully I can get a cogent response:
If the FBI had spent less time, money, and effort manufacturing cases (see Trevor Aaronson's The Terror Factory), harassing peace activists (see www.stopfbi.net), 'enforcing' websites' Terms of Service (see the prosecution of Aaron Swartz), and being hand-maidens to the MPAA (see the extra-territorial, and from all appearances extra-judicial) prosecution of Kim Dotcom, is it possible that they might have had some time, money, and effort to 'connect the dots'?
Didn't the Russians point a finger at these two?
Yeah, the Russian government warned us about Tamerlan Tsarnaev. But I'm sure that's just one dot in a million. I'm sure the Russians name ten of thousands of potential terrorists a year, no reason to give that any more weight than say a person buying fertilizer.
Even if the FBI spent all its time and effort on terrorism, that doesn't mean the job becomes easier. As Bruce pointed out, more hay on the haystack doesn't make finding the needle easier. It's entirely possible that without the extra data point of "these guys committed a terrorist attack" pegging them as terrorists wasn't really possible.
You're proving his point with the comment about the Russians. That seems like smoking gun style information after we already KNOW they're terrorists. But without that extra piece of information, connecting the dots might not be so easy. The Russians aren't magic terrorist detectors either, they have their own set of dots they're trying to connect, and passing those dots along to other countries doesn't necessarily make the problem any easier to solve.
I also wonder if the Monday morning quarterbacking isn't just about hindsight bias but also about people wanting to feel safe. Implicit in the argument that the government should have connected the dots is the belief that it must be possible to do so.
That seems like a comforting belief to have, as opposed to the alternative world where sometimes it might not be possible to find a terrorist before they attack. "Intelligence failures" like this poke holes in that comforting belief, which could be part of the source of the reaction.
Yes, yes, yes. This hits three or four nails right on the head. Somebody high up in the executive branch needs to hire Bruce.
But, but, but... the FBI just thwarted a "Terror Attack" in Montevideo, MN. It seems we have way too many 'terrorists' and not enough 'clowns with a couple pipe bombs' (or more accurately, 'with a couple suspected pipe bombs').
In hindsight, we know who the bad guys are. Before the fact, there are an enormous number of potential bad guys.
It's even worse than that, they aren't bad guys until they actually do something so there is no possible reliable indicator of potential badness until the act is in progress. People ask why the Tsarnaev brothers weren't picked up some time ago but the fact is that at that time they hadn't done anything wrong that would justify such measures. Unless the public is willing to inter and prison millions of people based on no evidence on the off chance that they might commit a crime, or alternately continue turning the country into the worlds largest open air prison (take that Austrailia!).
That doesn't even take into account the factor that enhanced security and military measures induce far more crime and violence in retaliation than they prevent.
Steve Jobs, one of the most successful business people in recent memory, further noted "You have to trust that the dots will somehow connect in your future."
This principle would serve us better than people seeking to blame others for political gain over perceived intelligence failures.
This whole "connect the dots" and "proactive" thing makes me think of an old buddy. We blew up all the things we legally could, and a few that we shouldn't have. He had a pal who had played with synthesizing nitroglycerin. We roughed out all manner of designs for IEDs, talked about where one might plant them for maximum effect. Disaffected? We both left college before graduating.
Obviously should have been on a watch list, maybe taken into custody just in case. And yet 30 years later, we're relatively productive unconvicted members of our communities. If the FBI had spent any time on us it would have been wasted and then some.
I agree with the premise of the article, but there comes a point where enough is enough. Since 2001, we've been subjected to grievous violations of privacy. America isn't America any more. I fully believe it's fair to criticize LE for not stopping this. If this was 1999, fine. But in 2013, after going through every citizens hind-quarters with fine toothed combs, every day...nice job LE. They deserve the criticism IMO. And if you think that by *not* criticizing them, that they won't look to grab even more power...good luck with that. Maybe criticizing them will actually wake people up and they won't be granted more power that won't help. Want to stop the criticism? Stop going through my email.
This article is a bit ironic coming from someone who is known to be able to decode a box of Alphabits :-)
It's become self-evident that the government is unable to connect the dots with these things. But surveillance is big business. Lobbyists love a surveillance state with unlimited funding, and by spending lots of money and building massive datacenters and collecting massive amounts of data, politicans can pretend to be "doing something."
In other words, data mining and monitoring and collection is going to keep happening whether it's effective or not.
Bruce, I think your comment "... think of it as a million unnumbered pictures superimposed on top of each other" does not go far enough.
It is more like pictures of the lives of every person living in this country at any particular moment (over 300 million). Reduced to dots. And then those dots overlaying the dots of the pictures of every other person that they've been in contact with or occupied the same space as or etc, etc, etc.
So they will never be able to "connect the dots" PRIOR to an attack. All they can do is reconstruct the person's life AFTER they've been identified.
You are correct to criticize the FBI and other agencies for manufacturing cases. It's theater. However, we don't even know for a fact that these two bombers were actually working on their own. It's possible, but there's a lot of evidence that suggests that they were patsies for the CIA, and other CIA contractors on scene. Were they? We may never know.
We just don't know what the hell happened, who did it, or why.
No matter what actually happened, the US media is certainly developing a narrative far from the truth with an agenda to continue the senior government's objectives (regardless of party).
Just read Chris Hedges interview with Julian Assange: http://www.truthdig.com/dig/item/...
What is reported and what is are almost always completely different things, and the most important things aren't reported at all.
The surveillance state is just the next stop in the course of empires. If you study the historical precedents, it is all completely predictable.
Most human information processing consist almost entirely of ignoring the right information (this is why image and audio compression works so well). The problem with needle-in-a-haystack problems, is that when the haystack is bigger than any one person can examine, our heuristics quit working. OTOH, once you know where to look, it's easy to hand someone (after the fact) a small haystack.
This is like finding a needle in a haystack?
In this case we are looking for which bad guy in a huge list of bad guys is about to do something that the media will care about. A more correct analogy would be:
This is like finding a needle in a needlestack.
Why do we hear nothing about the boys' uncle Ruslan? Other than his interview calling them losers.
He was married to the daughter of CIA biggie Graham Fuller. He registered a Chechan activist group using Fuller's home as the address.
He apparently worked for USAID in Kazakhstan.
His is currently a very well connected oil/gas lawyer.
Did he and/or Fuller pull strings to get the boys admitted to the US? Did they pull strings to get them US citizenship?
How did Uncle Ruslan become a citizen? Did he get any special treatment?
This seems to be something nobody wants to talk about.
Perhaps a bit off topic but the "needle in a haystack" analogy has long bothered me.
What is so hard about finding a needle in a haystack? Simply pass the hay over a magnet or a metal detector. Might be a bit time consuming but you could put a 3rd grader to work doing this.
And the needle would be found fairly quickly.
I do like the "needle in a needlestack" analogy that Harvey used. I assume that you meant a specific needle in a pile of similar needles. I'd not heard it before but am going to appropriate it.
It makes much more sense in this and any other context where the haystack metaphor might be used.
We need to revise the old chestnut; hindsight is not 20/20, it is 20/10. Or maybe 20/5.
I agree with your central point but you do not stress one fact enough, the fact that stories work. President Obama said that his number one regret of his first term was he had wished he had spent more time with telling stories rather than developing policy.
The battle between complex policy and simple stories is a losing battle for complex policy in a democracy where 50% of the population has an IQ below 100 . But that is OK. Because the truth is that the stories that we tell each other and which we tell ourselves are not inevitable. The fundamental problem is that we have let a "security narrative" dominate our society. It is the story we tell ourselves and each other. But there is a civil liberation narrative that can be told. The Americans who passed the Bill of Rights told that story. We can too.
You once said that poor hackers hack machines and great hackers hack people. Hacking people is more than just psychology. It is poets and novelists and screenwriters. It was Freud who said, "no matter where I have been analytically some poet has been there first."
The challenge that faces those who care about liberty is developing a new narrative of liberty. So long as civil liberties are always on the defense they will always fight from a weakened position and will more often than not lose. Let's stop bemoaning the calamity of hindsight bias and start telling our own stories, to ourselves and to each other. We should not be afraid of fighting on that battleground; we can win it.
Re: Finding a needle in a haystack, @John Henry.
The phrase dates back to the 1600s where they didn't have industrial magnets or metal detectors, and may not refer to a sewing needle - I've seen one statement of "finding a pine needle in a haystack". If it does refer to a sewing needle, it might be a handmade one, made from a splinter of wood or bone.
The expression hasn't aged well; the key is finding one special something in a large stack of similar-enough somethings. I think finding a needle in a needlestack might well be the best translation to modern ideas.
I want to add to the needle-haystack analogies:
The FBI is adding straw to the haystack - not the data. In fact, IMHO, they're building lots and lots of haystacks that they know are completely devoid of needles - just so they can get paid to search them.
If the FBI stopped doing this, they would have more eyes/hands/ears/brains working on that haystack - and they'd find that needle much quicker (and without the easy solution of burning the haystack down...).
It's not a sailboat... it's a schooner!
It's not a haystack. The Ruskies (NKVD) sent a warning to their counterparts at the FBI, and the Saudis sent a warning to State. Those should have intersected; that was the sole purpose for the creation of DHS. Once alerted, a cursory background check would have found other indicators: parents split with faith differences, non-mainstream demands put on suspect's wife, career/job at loose ends, financially unsuccessful, etc. In combination, those should have enabled a letter or search warrant for monitoring his computer.
In turn, sharing his (known now) Chechen connection back to NKVD could have obtained more details from them in trade. Same with Saud, the reason for his visa denial was never explored.
I just want to know who comes up with silly names like "Terrorist Identities Datamart Environment".
After the 9/11 disaster, we learned that a flight instructor had reported to the FAA and the FBI that some Arab men wanted passenger jet flying lessons but not landing lessons. We supposedly learned from this failure to act.
But, after the Boston bombing, we learn that the brothers had been questioned before, that the Russians informed us that the older brother was dangerous, and that the Saudis told us the same thing. When two nations, one of which we are not on good terms with, tell the US government that an immigrant is dangerous, then we should act. (There were grounds to deport the older brother.) This wasn't a situation of picking out a potential terrorist among millions of non-terrorists. It was a situation of completely ignoring evidence of criminal intent. It was inexcusable.
It's not finding a needle in a haystack, it's finding a particular blade of hay in a haystack. There are lots and lots of of "leads" and facts. The "security enhancing" actions proposed by politicians, like adding security cameras, only add more hay.
Like refusing to be terrorized, we just need to accept that preventing evil is science fiction (and usually dystopian SF at that). Nobody wants to prevent auto crashes with 5 MPH cars, and nobody really wants to spend what would be required to prevent terrorism.
It's not just about connecting the dots, it's also about sharing the dots. Giving one kid the entire page of a coloring book will allow him to connect the dots quite easily and find the animal that's hiding. If however you divide that one page among several kids, they will have to work together to literally figure out the bigger picture. As long as they don't, they're unlikely to find anything.
I believe many intelligence failures can or could have been avoided by TLA's working together and sharing information in a much more efficient way than is happening today. Unfortunately, most - if not all - are secretive little kingdoms with their own little princes, constantly fighting amongst each other over budgets, jurisdictions and esteem. The DHS only came about because everyone in Washington DC and beyond knew this was never gonna happen, so they decided to create yet another one to avoid the Kafkaian nightmare that would have ensued trying to align and integrate existing agencies.
As every engineer knows, technical solutions often are far easier to solve than the political issues that come with them, and throwing more resources at a problem not necessarily a better strategy than trying a different approach.
We're seeing hindsight bias playing out again now with the Cleveland kidnapping case. TV commenters are breathlessly asking how local police could possibly have not known what was going, why they did not search the house (because he may have left a kid on school bus) or because he (may or may not) have covered up the house windows, or just because. It's all obvious in retrospect.
Haha just stop invading other countries and terrorism will be less of a problem. People get upset when their country/region is invaded by the US and they want revenge. It's kind of natural if you think about it.
@paul: Your case is far from unusual. I'd be hard pressed to find a chemistry nerd who didn't occasionally blow stuff up with homemade explosives in his youth. About half of my final chemistry class in college was that kind of guy - and the other half had an unhealthy interest in poisons (that probably decides whether they'll study chemistry or pharmacy). Police would be quite busy if they tried to monitor them all. And they might never find an actual terrorist; I can tell from most plots that they were pretty clueless about chemistry.
@ Paul Rennault
"If the FBI had spent less time, money, and effort manufacturing cases (see Trevor Aaronson's The Terror Factory), harassing peace activists (see www.stopfbi.net), 'enforcing' websites' Terms of Service (see the prosecution of Aaron Swartz), and being hand-maidens to the MPAA (see the extra-territorial, and from all appearances extra-judicial) prosecution of Kim Dotcom, is it possible that they might have had some time, money, and effort to 'connect the dots'?
Didn't the Russians point a finger at these two?"
That is what I am thinking. 700,000 people? Really?
I liked Bruce's statement about increasing the size of the haystack to look for needles, in this context.
They probably are doing what all such agencies have a tendency to do, and as they have a history of doing -- looking in New York for a coin they dropped in LA. Because they like NY better. :/
Like I've always told my conspiracy theorist friends: If you geometrically increase the number of dots and remove the sequential numbers, you can draw any picture you want.
One of those articles by Malcolm Gladwell that I do recommend concerns exactly this issue:
In it, he draws a distinction between puzzles and mysteries. He defines a puzzle as a situation where you lack information, and finding more of it is the key to solving the puzzle. A mystery, on the other hand, he defines as a situation where you already have plenty of information and the challenge is to make sense of it. In a mystery, it’s not lack of information you need to address, but the abundance of it; simply gathering more will achieve nothing but muddy the waters further.
I don't think numbering the dots is that much easier after an "event".
Even then, the dots can be numbered in many different ways. This is the source of many conspiracy theories. For a current example consider this "numbering" in Infowars: Tsarnaev Brothers had a CIA Connection.
With dot-relationships like this, how does one decide which are valid and which are irrelevant?
Some might dismiss this as, "Well, that's just conspiracy theorists." But the same problem also results in convictions of the innocent; both legally and in the public mind.
As a result, it seems to me that it is just as specious to say, "Well, we were able to connect the dots afterward..." as it is to complain about "not connecting" them beforehand.
@Joseph R Jones
"Like I've always told my conspiracy theorist friends: If you geometrically increase the number of dots and remove the sequential numbers, you can draw any picture you want."
Authorities are often the worst buyers into conspiracy theories.
Endless, ludicrous stories there.
Some actually true. Some not.
Usually there is huge over reactions from the basis of statistically small events with investigative agencies with Gestapo powers. Usually these statistically small events are genuinely scary.
So are the investigative agencies with Gestapo powers.
It's like missing and exploited kids. We developed sensor nets, gps shoes, bracelets and tracking interfaces. Wearable computer research made advances. Trouble with security is they go around it or are distracted and can't pay attention and attention must be paid. Make Trax. More government projects? Low income housing is growing. The reverse Darwinism crowd is funded.
Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc.