Collecting Expert Predictions about Terrorist Attacks

John Mueller has been collecting them:

Some 116 of these Very People were surveyed in 2006 by Foreign Policy magazine in a joint project with the Center for America Progress. The magazine stressed that its survey drew from the “highest echelons of America’s foreign policy establishment” and included the occasional secretary of state and national security adviser, as well as top military commanders, seasoned members of the intelligence community, and academics and journalists of the most “distinguished” nature. Over three-quarters of them had been in government service, 41 percent for over ten years. The musings of this group, it was proposed, could provide “definitive conclusions” about the global war on terror.

The Very People were asked to put forward their considered opinions about how likely it was that “a terrorist attack on the scale of 9/11” would again occur in the United States by the end of 2011—that is, by last Saturday.

Fully 70 percent found it likely and another 9 percent proclaimed it to be certain. Only 21 percent, correctly as we now know, considered it unlikely.

I’ve never heard this particular quote before, and find it particularly profound:

In 2004, Russell Seitz plausibly proposed that “9/11 could join the Trojan Horse and Pearl Harbor among stratagems so uniquely surprising that their very success precludes their repetition”….

More predictions here.

Tags: , ,

Posted on January 10, 2012 at 6:56 AM • 28 Comments

Comments

Rookie • January 10, 2012 7:55 AM

It’s easy to find fault with people who have the job of trying to predict the future, whether it’s the weather, sports outcomes, or terrorism. Of course, they still need to be held accountable when their predictions drive public policy, but it’s an inexact science by definition.

What I would find more interesting is an analysis on why their predictions where wrong and there were no significant, coordinated terrorist attacks on US soil since 2011. I can’t believe it’s because they don’t want to strike the US. Are the US intelligence agencies doing so well at interdicting the plans and the planners? Have the wars successfully carried the terrorist fight to their front door? Are the terrorist group’s capabilities that degraded?

I have my opinions, but I think it would be valuable to have a definitive answer to that question to help drive policy for the next 10 years.

Nobodyspecial • January 10, 2012 8:00 AM

Actually if the Trojans had the TSA – the Horse would have worked again. While of course banning any wooden horses, pictures of horses, mention of horses and country-and-western songs.
They would have happily dragged any giant wooden hippos, elephants and rabbits into the city.

Graham • January 10, 2012 8:02 AM

Of course, it is possible for something to be “likely” in terms of having a high probability, and then for it to not happen, so for all we know the 70% could have been correct. All we can definitely claim is that reality was inconsistent with the view of the 9% who predicted certainty.

Michael Toecker • January 10, 2012 8:07 AM

I wonder if Stuxnet would fall into the category of “infamous, not repeatable” or if we haven’t learned the lesson yet.

Mike Toecker

Mark R • January 10, 2012 8:24 AM

I heard a talk recently about the perceived vs. actual value of experience in analytical fields. I apologize for not having any link to the study, but one specific example centered on polygraph testers. Experts in the field cite numerous techniques for identifying when a person is lying (tics, mannerisms, etc.).

The study found that years of experience did not significantly improve the testers’ ability to identify dishonest statements. They did, however, significantly increase the testers’ confidence in their assessments.

A further finding was that “asking a panel of experts” is a remarkably bad way of arriving at correct answers. Better results can be obtained by asking a single expert before all the experts have a chance to talk to each other.

Clive Robinson • January 10, 2012 8:45 AM

@ Bruce,

Now you have some time on your hands before the “book signing” rounds….

You could have a look back at what posters to this blog have said in the way of the same predictions, and also where they have given their reasoning behind it, and the accuracy of their predictions.

I’ve made a fairly consistant point that as long as the US are fighting on the home soil of various countries, 9/11 style attacks would not be attempted. I’ve also predicted that there would be more Richard “hotfoot” Ried style attacks that would achieve little or no harm but plenty of publicity to keep money roling into the terrorist coffers. And it kind of turned out that way including the “laser printer bombs”.

And I’m guessing that due to the Olympics being in London this year, we can expect to see some form of “attention grabbing” activities be it from the terrorist or from the authorities of which the latter I think will be an order or two of magnitude more likely.

What I’m awaiting on is what will happen now the US has pulled out of certain territorial areas. If nothing else had changed I would expect more terrorist activity on or near US territory. However contrary to the way it is presented in the news, various political interests are starting to engage with the political elements of various “terrorist” organisations and I suspect that this “recognition” unless badly mismanaged will provide a limiting effect in the short term.

This is important because “Arab Spring” has probably had more of an effect on the terrorist organisations than we give credit for. In the long term the current “arab terrorist” organisations days are effectivly numbered as the days of the opressive regimes are ending, and their only hope to survive with influence is to gain political representation before the time other interests get political power.

Brandioch Conner • January 10, 2012 9:21 AM

The only proof is prediction. If you cannot accurately predict, then you do not understand the situation.

@Rookie
“What I would find more interesting is an analysis on why their predictions where wrong and there were no significant, coordinated terrorist attacks on US soil since 2011.”

Because terrorists are extremely rare.

Terrorists who can operate in the USofA are even rarer. (Think about your ability to operate in Iran. Can you read the language? Speak the language? Tell a cop from a crossing-guard?)

Terrorists who can operate in the USofA while keeping their mission a secret long enough get the supplies needed to carry it out are the rarest of all. They are so rare that no practical predictions can be made about them.

Which is why I agree with Bruce on spending the money devoted to “fighting terrorism” on areas such as emergency response and police investigations. At least that will have some outside benefits.

No One • January 10, 2012 9:56 AM

Michael Josem: It’s more like looking at the 10-day forecast from ten top meteorologists, having seven of them predict 70% chance of rain, one predict 100% chance of rain and two predict <30% chance of rain and then there being no rain and only two or three vaguely threatening clouds.

No One • January 10, 2012 9:57 AM

<30% chance of rain and then there being no rain and only two or three vaguely threatening clouds.

(Open angle bracket ate the end of my first comment.)

Bryan • January 10, 2012 10:29 AM

Some time ago I read a book by Steven Schnaars called Megamistakes. I think he found that only 21% of forecasts become “true”. So, it looks like expert predictions are in the ball-park.

Daniel • January 10, 2012 10:41 AM

With all due respect to the cognoscenti I said what Russel Seitz said on 9/12/2001. Despite all the hoopla OBL did not exploit a security vulnerability; he exploited a psycho-cultural vulnerability. In fact, that vulnerability window in the American psyche didn’t even last an entire day as flight 93 proved. That was my first realization. My second realization took a few more days to form which was that no one would care and we would spend billions of dollars fixing a problem that would never reoccur. My third realization which came about a mouth later was that I was too “nice” (read:
weak) to join in the feeding frenzy. 🙁

In hindsight my one big mistake was that while I got the direction right I totally underestimated the magnitude of the effect. In wasn’t until about three years later that I realized that the 9/11 overreaction was no passing fancy but rather a generational problem and that rather than billions of dollars it would be trillions.

echowit • January 10, 2012 10:51 AM

What might be interesting would be a comparison of the severity/certainty of their predections and their subsequent “popularity” (rise or failure to rise) within the bureaucracy/hierarchy of their respective fields.

Any guesses?

Snarki, child of Loki • January 10, 2012 12:04 PM

You can play the same game over the next 10 years, and guess what?

Most likely, your prediction will be WRONG.

Doesn’t matter whether you predict big attacks, no attacks, whatever: most likely WRONG.

So the only real question for those making public prediction is: what type of wrong prediction does your career/reputation more good or harm?

Sure, you might get “lucky” and make an accurate prediction, but it would be foolish to stake your career on it.

Anton • January 10, 2012 2:07 PM

‘In 2004, Russell Seitz plausibly proposed that “9/11 could join the Trojan Horse and Pearl Harbor among stratagems so uniquely surprising that their very success precludes their repetition”….’

This is more like a definition of which we have 3 instances already. Interesting will be what shape and form the next one will take.

Figureitout • January 10, 2012 2:32 PM

Wonder what their predictions are regarding:

The next war?
Economic martial law?
A “bank holiday”?
The closing of the Strait of Hormuz?
Oil prices?

I imagine a scenario where the input into their modeling software caused the supercomputer to simply explode…

ted • January 10, 2012 3:34 PM

I have to wonder if the scarier predictions were made for no other reason so that laws like the patriot act could be passed with little opposition or to justify NSA eavesdropping.

Brett O • January 10, 2012 3:54 PM

  1. You cant predict what you cant define (terrorist, terrorism, etc.)
  2. terrorism is an act against the psyche (tip to Daniel – hit it on the head)
  3. predictions, such as weather forecasting, is no longer in the realm of meterologists gut feeling – its mega-computing using a hundred years of data points and hyper-sophisticated modeling applications; predicting terrorism cant reach that since the data generators are not observable, there is no history and few if any data points are being collected
  4. the government’s resulting terrorism follies is a historically based reaction to a non-historical (terrorism) activity – create a new agency to fight a new threat that uses a Cold War model – “Global War on Terrorism” anyone? War and terrorism are unrelated-tanks stop armies, not terrorists.

Dave • January 10, 2012 5:58 PM

@Graham:

Came here to say roughly the same thing, but I think you worded too weakly – we know that the 9% who claimed certainty where wrong, yes. We don’t know that those who said it was likely were wrong – but we do know that they were probably wrong.

S Taylor • January 10, 2012 7:39 PM

Simply put, the predictions were a function of FUD. I bet we hear some of this kind of FUD after the conventions when the presidential campaigns are in full swing.

NobodySpecial • January 10, 2012 8:28 PM

Even more ironic, the attack on Pearl Harbour was an almost exact copy of the British navy’s sinking of the Italian fleet in Taranto almost exactly one year earlier.

This was the first use of carrier borne aircraft attack on another fleet and was carefully studied by the Japanese navy – although unfortunately not by the US navy.

cagey_C • January 11, 2012 2:22 PM

While all three listed “stratagems” were tactically successful, only the Trojan Horse brought about the strategic victory the attackers sought. One could argue that the other two (Pearl Harbor and 9/11) were actually strategic blunders.

EIA • January 11, 2012 6:13 PM

..something about looking hard enough for something, and it being found – whether it’s there or not, or in between. If that fails, perhaps manufacture it. Last resort, make a soap-opera, or sign NDAA.

http://tinyurl.com/7tjt736 – ?

With “Homeland Security” now part of nearly every university’s curriculum, terror should be quite predictable.

johndburger • January 12, 2012 10:39 AM

By Mueller’s reasoning, the experts who said “unlikely” were also wrong, since “no chance” was a possible choice on the poll. The actual post facto probability being zero, the only correct choice was “no chance”.

This whole thing just illustrates how profoundly wrong-headed these kinds of formulations are.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.