Schneier on Security

Carlo Graziani • April 17, 2012 7:41 AM

I like that paper. A few comments:

(1) It’s too bad that they conflate “weird evidence or data” with “weird hypotheses” under the same term, “outliers”. The two should be kept logically and terminologically distinct, to avoid risk of obscuring the important distinction between evidence and conclusions.

(2) I really liked this quote about “foxes”, who “…thought very differently… they had no template. Instead, they drew information and ideas from multiple sources and sought to synthesize it. They were selfcritical, always questioning whether what they believed to be true really was. And when they were shown they had made mistakes, they didn’t try to minimize, hedge, or evade. They simply acknowledged they were wrong and adjusted their thinking accordingly. Most of all, these experts were comfortable seeing the world as complex and uncertain — so comfortable that they tended to doubt the ability of anyone to predict the future. That resulted in a paradox: the experts who were more accurate than others tended to be much less confident they were right.”

This is a perfect description of the mindset of a successful scientist. It is interesting that the same mindset is the ideal of what an intelligence analyst should aspire to.

(3) The authors discuss the “groupthink” phenomenon that has been rehashed ad nauseam in connection with the 2002-2003 analyses of Iraqi WMD. I really dislike and deplore this formulation, because it puts the onus of the failure entirely on the intelligence community, and disregards the very clear effect of the policy principals — from Cheney and Rumsfeld on down — who already knew the answers they expected the IC to provide, and were prepared to discredit any analysis — and punish any analyst — contradicting those answers. To the extent that an atmosphere could have existed in the IC that considered outlier opinion, it was systematically poisoned by the political supervision of that community. The responsibility for this poisoning is omitted, perhaps deliberately, in discussions of the analysis failure that focus on community “groupthink”.

Carlo Graziani • April 17, 2012 10:03 AM

@Gweihir: I would hope that whatever the intellectual failures of intelligence analysts may be, they don’t fall into the category of Dunning-Kruger-style incompetence. The failures of their political masters may be another story, though.

One reason I’m interested in the analogy to the scientific mindset is that outlier data and outlier hypotheses play an important, and even famous, sociological role in science: Thomas Kuhn’s paradigm shifts. These are cataclysmic changes in scientific models driven sometimes by data that doesn’t fit existing models, sometimes by models that groan under the burden of internal inconsistencies.

Typically, Kuhn argued, it takes a while for the community to perceive the implications of the outlier data, or to realize the necessity of substituting an outlier model for the existing defective one (or even to realize the existence and importance of the defects). Eventually, however, a new set of ideas sweeps away or subsumes the old ones, and the world suddenly looks totally different to its students, despite the fact that the data themselves are unchanged — the “paradigm shift”.

There seems to me to be a useful potential connection between the sociology of scientific revolutions and that of the assimilation or rejection of outlier data and hypotheses in intelligence analysis. In both cases, there is consensus on what counts as the principal data, and on which principal hypotheses are most likely to explain that data. In both cases a potentially revolutionary role is played by the intrusion of other, apparently inconsistent data, and of inconsistencies in the stories told to explain the data. And in both cases, it is necessary for the intellectuals at work to consider rejecting part or all of cherished, firmly-entrenched world-views, in order to bring about an intelligible new world-view that is more coherent and better capable of assimilating the available data.

That sounds like another paper, though.