Schneier on Security

AlanS • January 11, 2016 2:40 PM

On reading Rapp’s discussion I don’t find his use of Vaughan’s notion of normalized deviance very insightful into how the accident he describes happened. What she describes and what he describes are completely different. There’s a lot of interesting and subtle social science going on in Vaughan’s writing that just seems to go over his head because he doesn’t appear to be familiar with the literature she references. He’s a pilot; not a sociologist.

These statements are at odds with Vaughan’s analysis:
Rapp: the pilots were engaged in a “litany of errors and malfeasance”
Schneier: “flagrant violations of procedure become normal”
Banja: “The rules are stupid and inefficient!”
And you won’t stop the process Vaughan describes by:
Rapp: “a willingness to confront such deviance when it is seen, lest it metastasize” because it is only ‘deviance’ in hindsight.

The whole point of Vaughan’s analysis of the Challenger disaster is that it is a critique of the official reports of supposed behavioral malfeasance and the assignment of blame. It is easy to tell a story of malfeasance when you know the outcome. If you go back and immerse yourself in the sociology of the situations leading up to the event, suspending what you already know, then things might look rather more complex and difficult.

For those interested in the sociology I suggest reading the following account that covers the NASA study and two others: Diane Vaughan 2002 Signals and Interpretive Work: The Role of Culture in a Theory of Practical Action (PDF).

This is not a description of “malfeasance” or “flagrant violations of procedure”:

Analogical with the pattern in uncoupling, as incidents occurred the managers and engineers did not define the technical design as a serious problem because the social context and patterns of information affected interpretive work: As decisions were being made, signals of potential danger appeared mixed, weak, and routine. Mixed signals were information indicating something was wrong, followed by information indicating all was well. For example, the first incident of a technical anomaly on the Solid Rocket Boosters (a signal of potential danger) was examined, the cause identified, and the problem corrected. Then for five subsequent flights, there were no anomalies on the boosters (signals that all was well). A weak signal was one that at the time had no apparent clear and direct connection to risk and potential danger, or one that occurred once but the conditions that created it were viewed as rare and unlikely to occur again. Routine signals were anomalies that occurred repeatedly, but were expected and predicted as a consequence of a new safety procedure, and that engineering analysis supported as tolerable. Analogically, like the person left behind in an intimate relationship, only after the disaster were the managers and engineers able to look back and see clearly the meaning of the signals of potential danger that were there all along. (emphasis added)

Technological uncertainty created a situation where having problems and anomalies on the shuttle was itself a taken-for-granted aspect of NASA culture. The shuttle technology was of unprecedented design, so technical deviations from performance predictions were expected. Also, the forces of the environment on the vehicle in flight were unpredictable, so anomalies on returning space flights were frequent on every part on every mission and therefore routine and normal. Within this context of taken-for-granted problems, having problems on the SRBs was not a deviant event: Problems were normal and expected. Changes in the quality and quantity of the problems on the SRBs occurred over several years, introduced gradually into a cultural context where problems and change were taken-for-granted. Had all the damage to the boosters occurred on one mission, or on a series of missions in close succession, the sudden change might have been the attention-getting strong signal necessary to produce an official redefinition of the situation that the booster design was not an “Acceptable Risk.” As it was, signals of potential danger occurred in an ongoing stream of problems that tended to obscure change. History and experience mattered to the frame of reference the work group brought to the interpretation of information in a second way. The engineering rationale developed to justify the first anomaly became the precedent for accepting anomalies in the future. That first engineering risk assessment was foundational, for the first technical analysis was elaborated in greater detail with tests and analysis each time, so that the past and past decisions became the basis for subsequent analysis and launch decision making. The technical rationale for launching with anomalies was reinforced and made stronger in the process. (emphasis added)

Note the similarity to Kuhn’s analysis in The Structure of Scientific Revolutions: the interpretative work around anomalies, the elaboration of the community’s accepted problem solutions, etc. 

AlanS • January 11, 2016 8:55 PM

more from an interview with Diane Vaughan:

The Challenger case is a good example. Initially, it appeared to bea case of individuals – NASA managers – undercompetitive pressure who violated rules about launching the shuttle in order to meet the launch schedule.It was the violation of the rules in pursuit of organization goals that made it seem like miconduct to me. But after getting deeper into the data, it turned out the managers had not violated rules at all, but had actually conformed to all NASA requirements. After analysis I realized that people conformed to « other rules » than the regular procedures. They were conforming to the agency’s need to meet schedules, engineering rules about how to make decisions about risk. These rules were about what were acceptable risks for the technologies of space flight. We discovered that they could set-up the rules that conformed to the basic engineering principles that allowed them to acept more and more risk. So they established a social normalization of the deviance, meaning once they accepted the first technical anomaly, they continued to accept more and more with each launch. It was not deviant to them. In their view, they were conforming to engineering and organizational principles. That was the big [discovery]. I concluded it was mistake, not misconduct. (emphasis added)

But note that on any activity there are on-going events that require judgement as to whether they can be explained and incorporated into routine knowledge/practice or they are real anomalies that require a major rethink and new practices. The judgements are embedded in the local culture, social relationships, practices, experiences, etc. There’s no magic, super-duper algorithm that’s going to tell you how to ‘objectively’ interpret an event as something that’s routine versus something that isn’t. There’s uncertainty. Communities are quite conservative so tend to stick with what has worked in the past and to minimize the adoption of dramatic changes in existing practices. Those sort of dramatic deviations from normal practice are themselves usually risky. As Vaughan observes, the O Ring issues were spread out over years rather than bunched in a few flights one after the other that might have heightened the perception that there was something involved that couldn’t readily be assimilated to existing knowledge/practices and that the threshold of unacceptable risk had been crossed.