Schneier on Security

Good idea for an essay.

> New York in 2006 is different from
> Kenya circa 100,000 BC.

Your dates need corrections - 2006 sounds so last year, and 100,000 BC was well before God created Adam.

Page 8:"But when faced with a loss, most people (70%) chose Alternative C (the risky loss) over Alternative D (the sure loss)."

Wouldn't Alternative C be considered the sure loss?

How good are humans at deciding "Should I eat this fruit of knowledge or not?"

The essay's discussion of the endowment effect was very interesting. How big an impact does this effect have in something like the stock market? If the sellers' estimate of something's value tends to be higher than the buyers' estimate, which one does the actual market value end up closer to? My guess is that the market value leans towards the buyers' estimate if there are more buyers, and the sellers' estimate if there are more sellers. But I know next to nothing about economics, and eagerly await someone knowledgeable in the field to tell me what really happens.

"Every living thing makes security trade-offs, mostly as a species -- evolving this way instead of that way -- but also as individuals. Imagine a rabbit sitting in a field..."

sounds like security trade-offs or evolution "for the good of the species" the popular yet inaccurate meme from TV nature programs. Correct me if I'm wrong but doesn't practically all (natural) selection happen at the level of the individual, as your example illustrates?

Bruce,

Suggestion: consider a bit of anthropology. The neuroscience angle is easy to push too far, given the state of the field and the flexibility of the human mind. Jared Diamond's "Collapse" has some good examples of historical ecological trade-offs, even if his limited anthropological background doesn't give him the analytical tools necessary.

For example, you have the Norse Greenlanders starving a few miles from succesful Inuits, because of a cultural revulsion to eating fish. I'd expect that a number of our security illusions aren't hard-wired, but trained from millenia of living under agricultural serf/lord conditions -- not good training for information age conditions.

Prospect Theory: You mixed up C and D:
" * Alternative C: A sure loss of $500.
* Alternative D: A 50% chance of losing $1,000. "

"...most people (70%) chose Alternative C (the risky loss) over Alternative D (the sure loss)."

It is interesting to compare perceived risk to actual risk. It might be even more interesting to compare the differences in perceptions of risk among all the people, relative to actual risk.

Actual risk might be a mythical number (or at least an unattainable one). When we talk about real risk vs perceived risk, we are probably actually talking about two perceived risks, both of which vary substantially from the unknown actual risk.

Public policy needs to manage the varying levels of perceived risk across the population. When those perceptions are too widely divergent, policy makers will find their jobs much more difficult and maybe impossible. Rather than implementing solutions they may find themselves in a stalemate. Until people feel safe, public policy has not finished its job.

Public policy also needs to be concerned with actual risk. Maybe the most important consideration would be to make sure that the consensus perceived risk (whatever that is...) is at least as high as the actual risk. Addressing the perceived risk then becomes the focus for public policy.

Anyway, this seems like an important part of the discussion of perceived versus actual risk.

@k:

Natural selection happens at the level of the individual, but species improve rather than individuals. (Yes, this is an oversimplification.)

The rabbit doesn't change its mind. If it gets away from the predator, it doesn't need to. If not, it can't.

However, the rabbit population as a whole will get better at evading predators, because the less effective rabbits will be removed more than the more effective rabbits. This doesn't happen because of some mysterious force, but it improves the species just the same.

It's pretty widely recognised that the utility function for money is not linear, and so some of your comments on the utility trade-offs for sure and probable gains and losses could do with some tidying up.

There are a couple of points I think need more attention. The first is intentionality, which you address in your table but not really in the text. People are much more worried by risks from people who intend them harm (e.g. terrorists) than from those who don't (e.g. careless drivers).

Second, and related, is that our brains are hardwired to detect cheating, and to get upset about it. If you rephrase a problem in such a way that a correct answer will involve detecting social cheating (e.g. under-age drinking), then people are much more likely to get it right. There's a reference here: . I'd be pretty certain that any risk involving someone cheating so as to get an advantage they're not "entitled" to will be perceived as more serious.

@T

Natural selection occurs primarily at the level of the gene. The individual is ephemeral, the information in genes is eternal. Otherwise, you wouldn't see kin selection, and therefore no super-organisms like ants, bees and nake mole rats.

You would be correct to say that the evidence for group selection is still rather weak, that the primary bottle-neck for information propagation is the individual; that is, however, quite different from say that the unit of selection is the individual.

Sorry, the reference got mangled. http://heuristics.behaviouralfinance.net/Mont02.pdf

"Prospect Theory: You mixed up C and D."

Thanks. I will fix.

Typo: "Stated" should be "Started", page 12, third paragraph.

BTW: In my humble opinion, it's cheating to consider words that start with "Unk", like unknown and unkempt, to be words with K in the third position. Not that that has anything to do with the statistic. :)

Your Innumeracy section is underdeveloped; I'd either develop it more, or strike it.

As this is a long essay, I'd tend to re-cap the conclusions you make about security at the end of a number of your selections near the end of the essay, perhaps in its own section prior to "Making Sense of the Perception of Security" (or at the beginning of that section).

My personal issue (which may not be common in your audience) was that since I have a significant statistics and Psychology background, I found a lot of the groundwork material repetitive, and had difficulty separating out the individual points you were trying to make about security from that groundwork. If you think that your audience is unlikely to have such background, you can safely ignore this point.

@Matthew Skala
>How good are humans at deciding "Should I eat this fruit of knowledge or not?"

They were fine until a serpent got through the firewall with a phishing attack.

At least on my computer, there's something strange about the O in the word "two-man" on page 14, paragraph 4 (it looks like a different font).

I feel like a secretary :)

Page 14, paragraph 1 (first indent): "Linda is a 31 years old, single, outspoken, and very bright."

Should either be

"Linda is 31 years old, single, outspoken, and very bright."

or

"Linda is a 31 year old female, single, outspoken, and very bright."

And you forgot to mention if she is looking for a boyfriend.

@Alan: there is no "Actual" risk. Things either happen or they don't. We don't know which.

A probability is a useful number generated from a model built from the knowledge we do have. If we integrate all our knowledge into the model the number is as good as it can be.

That depends on what knowledge you have (or can afford the effort to integrate): I have just tossed a coin. What is the probability that it is heads?

For you, 50%. For me it is not. Our knowledge is different, therefore so are our probabilities.

(Frequent events have a frequency, which has an important relationship to the probability of the event in a short unit time. The large number of events have given us a lot of knowledge to build the model on. This might be confused as "actual" risk).

Very interesting discussion, with some good comments already.

One of the things that horribly complicates this discussion in the modern world is the use of media to give everything a spin. You can now convince very large groups of people that the real threat is not what it is (it is something different from reality), and can do the same thing for percieved threats. This makes both of them that much harder for the "common person" to measure, as this now has the ability to alter all of the standard heuristics when taken outside of the lab. While there have been good materials and studies on the use of media and media manipulation, I suspect this field (the use of media to influence security decisions, etc.) is very much in its infancy. You may be too far ahead of the curve on this one, but perhaps the discussion alone is what's needed to get the ball rolling.

This use of media to alter perception specifically in regards to security issues is a good part of the basis for the "Knowing Your Enemy" essay. It's also kind of the basis, I think, for "Wag The Dog", though perhaps in a different way.

As a side note, it's a shame Turner Broadcasting caved. They are telling everyone in the US that stupidity is acceptable, and the costs for said stupidity will be covered. A very bad precedent. "If I give you a sandwich" indeed!

Rainer

@T:

I agree with your points.

My post had more to do with bruce's contrast of security trade-offs made "mostly as a species..." vs. "but also as individuals...."

If the rabbit example is an example of an individual security trade-off, which eventually becomes prevalent in the species as a whole, then what would be an example of those tradeoffs (presumably the majority) "made ... as species"?

I'd just get rid of the whole "made mostly as a species" and the initial reference to evolution.

Suggest: (as long as were suggesting.)
"Every living thing makes security trade-offs. Imagine a ..." Then I think the discussion of consequences for human behavior follow.

The psychometric and economic models of understanding risk are undoubtedly critical, but I think they do not fully represent the "whole story". Studying risk has a long history in sociology and geography (under hazards research), and help us to understand the way society contributes to actual risks and risk perceptions.

I suggest looking for "social amplification of risk framework" if you decide to look into if further. For a current overview of the topic, I would also suggest Taylor-Gooby, P. (2006) "Current directions in risk research: new developments in psychology and sociology" Risk Analysis 26(2):397-411.

keep up the good word.

Typo: delete forth, insert fourth, bottom of page 1.

@Mark Smith, or at least write the article backwards.

oh, I feel like such an old programmer....

@Rainer

Good point, did you also notice that the american media didn't initially show pictures of the cartoon LEDs? It was hard to find when the story initially broke.

The foreign press displayed it almost immediately. Only later did *some* of the American media websites show a photo of the cartoon character.

Baby nit:

You say 1/250th:

Subjects tended to prefer ideographs they saw after the happy face, even though the face was flashed for 1/250th of a second and they had no conscious memory of seeing it. That's the affect heuristic in action.

But the source says 1/100th:

The shutter speed for the subliminal prime was set at 4msec which, after adding open and shut delay, results in a 10msec flash.

Figure 1 in the source also talks of "Subliminal 10 milliseconds"

Really enjoyed reading the article. A couple word edits:

evolutionary -> evolutionarily, before footnote 18

tw0 -> two, near footnote 36

'pari-mutuel' near footnote 46 is correct, but (IMO) clearer without the hyphen.

Page 13 bullet point 2: "One his way out the door," should be "On his way out the door,"

BTW: great read so. Thanks for sharing it.

It would be interesting to see the theory applied to modern advertising. And to the marketing of War on Terr'sm.

From the conclusion of Bruce's essay:
----------
In the past I've criticized palliative security measures that only make people feel more secure as "security theater." But used correctly, they can be a way of raising our feeling of security to more closely match the reality of security.
[...]
But used in conjunction with real security, a bit of well-placed security theater might be exactly what we need to both be and feel more secure.
----------

That is an intriguing conclusion, and one that I think politicians intuitively get. It's sort of a marketing approach -- the branding of a product needs to set a user's expectations for that product. "Theater" can therefore send the message "we are aware of your concern and we are taking care of it."

There is a (small) percentage of us who are allergic to manipulation of this kind. The advertising age has taught us to be mistrustful of the marketing messages that we receive. We read Consumer Reports instead of Motor Week. We want the real facts, not the glossy commercials.

But the issues described in the essay are not only for use in explaining why John Q. Public does a poor job of risk assessment, or how to make him feel comforted at the airport. This is also a checklist for how to make objective cost-benefit decisions in public policy -- i.e. "real security".

So if a risk's SPCET (severity/probability/cost/effectiveness/trade-offs) have been objectively considered in the making of public policy, thereby producing real security, then it may indeed also be appropriate to add some "theater" dressing on the policy, to comfort human nature.

On the other hand, if the public doesn't "get" the SPCET of an issue at the outset, their elected representatives are less likely to actually produce rationally-based policy.

Here's to hoping that more objectivity goes into the decision making process, and that it's professional advocates educate the public at large as well as the politicians.

PS - Bruce, I spotted a couple of typos in the draft. In one place, "stated" instead of "started" was used. In another, you wrote "...that I mean" instead of "...by that I mean" (or something similar). HTH.

Bruce, maybe you should consider vetting your articles on a wiki for easy editorial :)

Worth a read:
http://www.apple.com/hotnews/thoughtsonmusic/

(1) The science in this bit is pretty inaccurate: "It's what pumps adrenaline and other hormones into your bloodstream, triggering the fight-or-flight response, causing increased heart rate and beat force, increased muscle tension, and sweaty palms."

(2) The two clauses here don't fit together grammatically: "subjects didn't care whether they received $15 today or $60 in twelve months, while at the same time indifferent to receiving $250 today or $350 in twelve months, and $3,000 today or $4,000 in twelve months. "

But interesting essay.

tw0 instead of two: "Groups of six observers watched a tw0-man conversation from different vantage points"

There are several examples where I find that my own instincts run strongly with the experimental results and against the "rational" choice. I can't fault the argument for the rational choice but nonetheless would find it very hard to adopt it as my standard behaviour. Interesting.

Bruce, a sugustion. There is a something I have observed that I think you could give some valuable insight into.

Unfortunately I am not a psychologist so I do not know the correct terminology for what I am describing, however I shall try to be as clear as possible, and I'm sure someone will know the right jargon.

People have a tendency to assume something is better because it costs more. Most of the time the more expensive thing is better, but not always, as any smart shopper can tell you.

I think there is a similar effect at work when it comes to evaluating security at the "gut" level. People often think that the more restrictive (costly in terms of freedome and convinence) security measures are the more effective ones. Yes, everyone complains about draconian security, but at some level there is an intuitive notion that they are getting more security by paying more.

Also, It is easy for people to compare the cost they pay in time to go through different security protocols. It is much harder for them to compare the benefits of two different security protocols, because the chances of a real attack affecting a particular person is so so small.

Of course an overly-slow screening line is seen as inefficient, implying incompetent security. But at the same time, a screening line that moves "too fast" leads people to believe that the guards are just waiving them through without scrutiny.

I think this may explain why so much of the US has embraced particularly costly forms of security theater.

Mike Scott, "People are much more worried by risks from people who intend them harm (e.g. terrorists) than from those who don't (e.g. careless drivers).", may or may not be correct in general, but is certainly wrong in these cases.

A constant barrage of propaganda ensures fear of "terrorists". A more subtle but just as pervasive snow of reassurance gently presses us to discount traffic risks.

Evolution can be deceptive. We, 21st century Anglos, tend strongly to think about individuals, while evolution deals only with large numbers. Furthermore, without unscientific variation, evolution does not work at all.. the large numbers become predictable quickly become lunches.

Security experts, at least those who hire them, have no interest in increasing the likelyhood that the target population will survive. On the contrary, they would cheerfully eliminate the entire rest of the population to improve their own individual chances. In the wild, that individual would leave zero offspring.

Julius Caesar can choose to kill 200 or 600 Gauls, indeed to kill exactly 200. A (huge) pride of lions can make no such choice.. they will kill approximately 200 antelopes, maybe 201, maybe 199, rarely 202 or 198. The Gauls can attempt to influence Caesar. The antelopes cannot influence the lions, they can only run and jump randomly.

Indeed, they must jump somewhat randomly, even if a hotshot security expert could prove lions are incapable of dodging left. Should any significant fraction breed to become exclusive left-jumpers, the ability to right-jump (even sometimes) would be quickly eliminated from their genetic repertoir, along with the right-dodging lions. Then, either the antelopes would over-populate and starve, or some left-dodging predators would invade and eat them.

For evolution to have a happy out-come, security experts must be randomly disobeyed. A suitable number of predators must be fed, every day; and a larger number of prey must escape, every day.

Note, too, there are 200? lions and 200,000? antelopes. The security expert thinks there is only 1 Julius Caesar and 600 Gauls. Of course, not. However, the kinds of decision making suitable for evolution are nonsensical in the scenarios presented by security experts. Evolution preserves (the better) representative samples of populations. Security preserves non-representative individuals.

It occurs to me, having just written the previous sentence, that I ought to complete it with, "at the expense of the population".

Furthermore, to avoid the inevitable fate of losers in the game of evolution, the population ought to kill all the security experts.. first interrogating them aggressively to find out who hired them, so we can kill their bosses too.

Bruce,

Thoughts are:

1. This is hugely important work - but just scratching the surface. Therefore I would be careful about any conclusions at this time.

2. Who is the target audience?

3. I add my vote to "nurtur" side (learned social norms / sheep like tendency / fear of rejection); your focus might be too strong on the "nature" side.

For example; fear of exclusion from social acceptance is one immensely strong motivator, amongst many. The existence of intellectual understanding or rational thought does not guarantee it's application.

4. I think we must never forget that people are not just statistics or animals. A suicide bomber motivated by a sense of injustice is hard to fit into a darwinian or statistic social model.

I didn't find the Linda example convincing. When presented with options like:

A. Linda is a bank teller.
B. Linda is a bank teller and is active in the feminist movement.

then a mathematician might think "B is a subset of A, therefore A is at least as likely as B". But most people, trained in language rather than mathematics, will assume that the question is shorthand for:

A. Linda is a bank teller and is not active in the feminist movement.
B. Linda is a bank teller and is active in the feminist movement.

With those choices, B seems more likely. The assumption that a set of presented choices is intended to be mutually exclusive is pretty strong.

@ Benny

"If the sellers' estimate of something's value tends to be higher than the buyers' estimate, which one does the actual market value end up closer to? "

If the sellers's estimate is higher than the buyers' - then they would do well not to be sellers. Seems contradictory to want to be a seller if you estimate the price to go up.

Estimates don't really matter - only actual trading counts in determining stock price.

"My guess is that the market value leans towards the buyers' estimate if there are more buyers, and the sellers' estimate if there are more sellers."

On the stock market, every buyer must be matched with a seller. There can not be a purchase without a corresponding sale, and therefore there can not be more buyers than sellers or vice versa.

What matters is the quantity of money being offered. If buyers anticipate that the stock will go up, but too few others think that it will go down (i.e. buyers market) - then buyers must induce the others to part with their stock. To do so the buyers must increment their bid price upward until the market is cleared - until those who initially wanted to keep the stock change their minds and become sellers.

to summarize:

a) buyers pay a premium to convert holders into sellers.
b) sellers lose a premium to attract buyers.
c) sellers RENT out stock from holders and quickly sell it (b) with the intention of later re-buying it (a) and returning it to the holders [short selling].

The price of the stock goes up if the quantity of money performing (a) outmatches that of (b) and (c).

The price of the stock goes down if the quantity of money performing (b) and (c) outmatches that of (a).

I hope that helps.

"Humans have evolved a pair of heuristics that they apply in these trade-offs. The first is that a sure gain is better than a chance at a greater gain. ("A bird in the hand is better than two in the bush.") And the second is that a sure loss is worse than a chance at a greater loss. Of course, these are not rigid rules—given a choice between a sure $100 and a 50% chance at $1,000,000, only a fool would take the $100—but, all things being equal, they do affect how we make trade-offs."

All so true until the example - the presented options are very inequal also from mathematical point of view.
How about taking an example from lottery - gaining $1 surely instead of getting $million with the odds of 1/1000000 or less?

@Marko: "All so true until the example - the presented options are very inequal also from mathematical point of view.
How about taking an example from lottery - gaining $1 surely instead of getting $million with the odds of 1/1000000 or less?"

This is why one would use the expected (average) value: 1$ for sure = 1$ expected -- 1.000$ with a chance of 1/000 = 1$ expected
Fact is that people will tend towards the sure alternative if the expected value is equal. They will even tend towards the sure alternative if the expected value of the risky alternative is higher than the sure value.
The exact point where they would swing towards the risky alternative is individually different. The term is "risk-aversion" and economists use so-called risk-aversion-functions to modify expected values accordingly.
Problem is, that it is very hard to find a risk-aversion-function for a given individual.

@ Benny,

correction: ' and therefore there can not be more buyers than sellers or vice versa.'

That should be buys/sells not buyers/sellers. That mistake is so common - even professional investors make it a habit.

There can be, for example, 2 buyers and 100 sellers, yet the total amount of buy/sell contracts would be 100.

---

@ Bruce, @ Benny

The 'endowment effect' experiment is highly flawed.

From the draft, page 9:

'It’s called the “endowment effect,� and has been directly demonstrated in many experiments. In one half of a group of subjects were given a mug. Then, those who got a mug were asked the price at which they were willing to sell it, and those who didn’t get a mug were asked what price they were willing to offer for one. Economic utility theory predicts that both prices will be about the same, but in fact the median selling price was over twice the median offer.'

This experiment is flawed because there is NO economic transactions occurring at all. There is no buying/selling but merely ASKING prices. There is also absolutely no mention of the money that they would use in exchange. Such a parallel world scenario yields absolutely no insights.

Asking prices are useful in real life to initially figure out how many buyers there would be - but as every business knows you don't always get what you ask for, and therefore you have to lower it until enough are willing to buy it - even if it means you have to sell it at a loss.

Economic utility theory makes no prediction whatsoever that they would be about equal. All it states that if buying and selling were to occur b/w the mug holders and non-holders, a market price would quickly emerge as each would quickly discover who is offering the best deals. We can not know in advance how many would want to buy or sell in that experiment. However, if the experiment required that all must buy and sell - it should be noted that the price would be ZERO. Why it would be zero is left to the reader to figure out.

@Bruce: "Steven Johnson relates ... If you're a higher-order primate living in the jungle and you're attacked by a lion, it makes sense that you develop a lifelong fear of lions,"

Steven describes what many may recognise as a mini case of Post Traumatic Stress Disorder. Current thinking on the mechanisms of this differs slightly from your analysis. In lower animals the violent stressor plants a memory, but it needs *reinforcing* to become a memory that triggers a fight or flight response. It's not one lion attack but two, spaced apart. If the second lion attack doesn't take place soon enough the memory doesn't entrench itself in the same way.

In humans we think situations like this over again, restimulating the memory. The remembering can entrench the traumatic memory and turn it into a fight or flight trigger. That is what, in some people, can turn a one time violently stressfull event into a lifetime of autonomic irrational response to it - a situation now known as PTSD. It's also why counselling immediately after such an event may do more harm than good.

@quincunx: Spot on, except your last paragraph. I see no reason the price should be zero, at least for the assumptions as given.

@Paeniteo: The lottery is not an investment -- it is entertainment, a fee for the pleasure of sitting around discussing what you will do if you win. People pay a dollar into their workplace pool so they can then chat about it, not because they think it is a good investment.

Bruce,

Would it be more precise to describe Alternatives B and D as a 50/50 chance of gaining (B) or Losing (D) $1,000 or $0? That seems to be implied but stating it might make the point more clearly to someone considering this for the first time. I agree that for those of us in the choir it is already understood.

Bruce,

Fantastic article, I don't really have much more to add other than if you're going to start looking into psychiatry more you should definitely check out some of the magic/mentalist material out there. It's absolutely fascinating learning about how and why we are fooled.

@ Ben Liddicott

"Spot on, except your last paragraph. I see no reason the price should be zero, at least for the assumptions as given."

I should have been more accurate and said the price would tend to zero, not necessarily zero.

The reason for this is that if everyone is required to buy/sell, then it is no longer a market - it is an exogenous edict, that to be effective would have to be enforced at the point of a gun (if not physically then theoretically).

So imagine we have an equal number of mug holders and non-holders, at the end of the experiment their roles will be completely reversed, or else we will have a bunch of dead bodies.

The first mug holder steps up and tries to get the best offer. Since every non-holder knows that the holder will be shot, they will offer a zero price - since no matter what the holder will be compelled to give it up.

The only reason they would not offer zero is out of benevolence, not out of economic self interest.

The same process will be true for the rest. The point is that once you make a transaction an apriori requirement, you are no longer dealing with free agents - the holders will be more concerned with not getting shot as opposed to getting a good price.

@gfujimori

Thank you. That's an interesting observation you made, and one that could probably be reflected in many "news" stories. I don't actually watch American news, not living down there, so I got my first information about this from places like BoingBoing which, if I remember correctly, had a mock-up of one from the start.

Based on some other blog comments I've seen, others were not so fortunate, pre-empting their ability to make a valid (read: useful) evaluation of the unfolding events. By the time word got out that these things had been around for two weeks (and what they really looked like), these people were firmly in the "punish the offenders" camp.

So, "informed" media let me make a more realistic assessment of the situation immediately. This gave my entire attitude toward the unfolding events a very different spin than that of others watching only mainstream media. By knowingly or unknowingly limiting the information given, mainstream media helped bolster the "valid threat" scenario.

Rainer

"Similarly, it seems to be evolutionary better to risk a larger loss than to accept a smaller loss. There may be some benefit to this bias, but it may simply be adverse selection based on individual appetite for risk."

One plausible explanation is that this is because for an animal living on the razor's edge between starvation and reproduction (not uncommon since populations often expand until they are limited by food scarcity), a small or a large loss of food may actually be equally bad: both result in death. Therefore the best option is to risk everything for the chance of no loss at all.

Tiny minor edit: You write "Groups of six observers watched a tw0-man conversation". Replace tee-doubleyou-zero with tee-doubleyou-oh "tw0" -> "two".

I have just read "Stumbling on Happiness" by Daniel Gilbert which is a funny book on how humans make decisions and the faults that are built into the decision making process. I would recomend this as background reading for your essay.
My thoughts on Risk after reading this:
If I drive at 90 mph and don't die then I percieve it to be safe. Every consecutive day I achieve this without dieing makes driving fast safer in my perception.
However if the gap between instances is great enough my risk measuring system resets and each instance is as dangerous as the first.
From a highway emergency services point of view they will spend most of their working day dealing with the aftermath of accidents and so form a much worse view of the risk involved (but will associate the risk with drivers who are "worse" than they are and so continue to drive fast while condeming the public for doing so).

I bet that after reading more about brain heuristics, cognitive theory, general psychology, tacit-explicit knowledge conversion, Bruce winds up being a proponent of complex adaptive systems theory.

Really, this is a knowledge management problem in a specific domain (security). Getting people to absorb explicit information properly into their knowledge base is a topic of a staggeringly large number of research papers.

Bruce, you need to post your recent reading list on your blog.

Automobiles kill 40.000 a year, while commercial airplanes only kill a few hundred people a year.

That statistic is a bit about apples and oranges.

The airplane figure is a 100% external risk: when you take a seat on an aircraft, there is absolutely nothing that you can do to affect the probability of dying.

The 40.000 automobile deaths figure is composed of both external and self-imposed risk, as it includes speeding drivers that wrap themselves around a tree (self-imposed) as well as the kid that gets run over by a drunk driver (external).

For comparison, I think it would be interesting to learn the figures for self-imposed vs. external traffic deaths.

That doesn't change the fact that drivers accept the much larger self-imposed risk of driving more easily than the small external risk of flying, but it will allow us to specifically compare the external risk of driving with the external risk of flying.

@quincunx:

Your hypothetical situation verges on the bizarre.

I think the problem may be that you've introduced a cost into your model without acknowledging that it's a cost. One presumably values one's own life over just about any amount of money. Effectively you've turned a commodity into a hot potato that sellers will readily get rid of at any price.

If you redefine the system so that sellers who fail to sell or buyers who fail to buy are, say, fined $5, then I believe you'd find that prices stabilize right around the amount of the fine. Of course, that continues to ignore real-life complications -- the commodity in this case is provided free to the seller and has equal worth to buyer and seller. In fact, the commodity is worthless except in terms of avoiding the fine. What I'm getting at is that by imposing a condition like this you've actually overridden the primary point of the experiment, which is to allow the subjects to hypothesize the value of a commodity *in a vacuum*. And the sole insight is that, again, humans naturally value what they already have higher than what they might some day acquire. In other words, a mug in the hand is intuitively valued at approximately 200% of one in the bush. This actually has very little to do with market economics, it's purely a psychological insight.

@ Eingang

"Your hypothetical situation verges on the bizarre."

Poetic license and dramatic effect. Compelling someone to pay a fine is still compulsion. What happens when they refuse to pay? (OK, granted I don't suggest the experiment should turn into a hostile environment).

"If you redefine the system so that sellers who fail to sell or buyers who fail to buy are, say, fined $5, then I believe you'd find that prices stabilize right around the amount of the fine."

A sale price of zero is still an exchange.

Your setup doesn't work out. If the first buyer comes up to sell, everyone would still bid $0, and that buyer would still prefer to get nothing as opposed to paying a $5 fine - and the same holds true for everyone.

The game is still a forced market in favor of buyers no matter which way you slice it: theft or death.

Are you to have us believe that the actors would voluntarily pay $5 to avoid a $5 fine - if they can avoid the fine altogether and exchange for free?

"Of course, that continues to ignore real-life complications -- the commodity in this case is provided free to the seller and has equal worth to buyer and seller."

No, in real life exchanges occur precisely because the two parties to an exchange have a reverse subjective preference for a given item. There is no 'equal worth'.

You buy a good because you prefer a good to your money - and the seller prefers the money to the good. If there is a perceived equal worth of your money and the good - you would not take action, since there is no benefit and an actual time/effort cost in making the exchange.

"In fact, the commodity is worthless except in terms of avoiding the fine."

Well yes, still assuming my stipulation that an exchange must occur - you have just proven yourself why the price must then be zero, 'the commodity is worthless' if it must be exchanged.

If my stipulation does not hold - then the commodity is not worthless, it's a MUG! It has a use and it can be traded for something else.

"What I'm getting at is that by imposing a condition like this you've actually overridden the primary point of the experiment, which is to allow the subjects to hypothesize the value of a commodity *in a vacuum*"

Yes, but then it is entirely pointless and cannot be used to gauge values at all. Again, no insight. I can shout all the prices I want, and still there would be no external value for anyone to observe, because it's all in my head. Furthermore since it's all in my head, there is no reason why I couldn't lie - how would you REALLY know how much I valued something?

"And the sole insight is that, again, humans naturally value what they already have higher than what they might some day acquire."

Uhm, no, you see if that were the universal case there would be no trade at all. Everyone would value what they have higher than what they could get, and you couldn't get anything since no one is willing to part with theirs.

Theorizing prices in a vacuum eliminates the price system entirely.

To suggest that sometimes people value what they own over what they could get, is more accurate, but nontheless obvious. No insight here at all.

"This actually has very little to do with market economics, it's purely a psychological insight. "

Market economics incorporates many psychological insights - after all the economy consists of humans.

@Sanctimonymous: Nice one. I'm lol.

@Bruce: Great essay, seems to be a nice expansion on your last essay on risk. I'm trying to some of the people in my life to read this one. For some, the points you make are real eye openers.

For the past month or so, I've been working on a paper (for a class--no current ambition to publish it) on a similar topic: how groups of people make decisions. One of the biggest challenges, I think, is figuring out how to improve the collective decision-making process. You, and many of your readers, are passionate about security issues, but most of the rest of the world sees security, or the lack of it, as just one more thing in life they have to deal with. Yet through democratic and market processes, their preferences strongly affect how their societies choose to deal with security issues. Regardless of whether you believe legislators vote with their constituencies or with their contributors, those legislators face strong pressures to satisfy the desires of people who are making their security decisions based on heuristics. So, I'd suggest expanding your treatment of how to change things to improve the quality of the security decisions individuals make despite the distortions arising from the heuristics you catalog. Either that, or propose a mechanism by which society as a whole can make good security decisions even though many of the people in it might want less-than-optimal tradeoffs.

"Second, when considering security gains, they're more likely to accept an incremental gain than a chance at a larger gain"

In practice, this is the preference for patching up leaky systems, rather than replacing them with systems that work.

It explains much about the security market :)

"Either that, or propose a mechanism by which society as a whole can make good security decisions even though many of the people in it might want less-than-optimal tradeoffs."

To be read: Use force if necessary.

Good reads on unreasonable beliefs:

"Why People Believe Weird Things" by Michael Shermer

Addressess many of Bruce's points on fallacies in reasoning in greater depth.

"The Lucifer Principle" by Howard K. Bloom

Addresses people choosing actions on basis of group beliefs as opposed to personal safety (among many other things).

"To be read: Use force if necessary."

Or a representative democracy. Or a security czar. Or a standing advisory panel to committees in Congress/whatever other legislature you have. Or converting Homeland Security from a department to an appointed or elected board. :-)

There are probably a bunch of others approaches the creative minds on this forum could devise.

@ False Data

"Or a representative democracy. Or a security czar. Or a standing advisory panel to committees in Congress/whatever other legislature you have. Or converting Homeland Security from a department to an appointed or elected board."

Force. Force. Force. Force.

"There are probably a bunch of others approaches the creative minds on this forum could devise."

Perhaps they can, but it will inevitably come down to force (the state) or not (the market). Most will pick the first, because they incorrectly perceive security deficiencies to be part of the market, whereas few will pick the market because they correctly perceive that security deficiencies are the result of the accumulations of bad legislation, unsound monetary systems, and unnecessary gov institutions in general.

Study wisely!

---

@ Fraud Guy

""The Lucifer Principle" by Howard K. Bloom

Addresses people choosing actions on basis of group beliefs as opposed to personal safety (among many other things)."

Interesting. Never heard of that name for the principle, always known it as 'groupthink'.

Great piece. One tiny typo:

>The first group was told to imagine
>that they has spent $50 earlier in
>the week on tickets to a basketball game,

Should be "have spent" :)

Here's an amazing story about a security judgement failure:

http://news.independent.co.uk/uk/legal/article2245143.ece

Bruce, my compliment for a great paper on a complex topic.

Of many possible comments to this highly interesting field that will raise mysteries for a long time

a) You write that test are often done on students. Problems is that students are perhaps bad test characters because they have not yet positioned themselves with debts, kids, carriers and all the other sensitivities that make people feel vulnerable and thereby likely increasingly riskaverse.

b) You should in my view have a bit more attention on the learning capability of man. A child once burned learn to fear fire. It is a real threat even though the risk may be overestimated.

c) You clearly do not include technologies that can handle trade-offs such as Privacy Enhancing Technologies.

d) I would REALLY like to see studies on why DEVELOPERS and IT buyers tend to underestimate the threat they are to others while they perhaps overestimate the threat of others to themselves. That seems to be the root security problem today.

e) The concept of CHOICE and AUTONOMY would require the availability of services that also incorporated individual security and control (both real and the perception). We learn with spouses that if you VOLUNTARY give op control for no other reason that do show trust, you would build must stronger trust-relationships. But the difference is that you want your spouse to trust you, Why care if some service provider trust you? He trust his security that he get his money, trust in me is not relevant.

"Bruce, maybe you should consider vetting your articles on a wiki for easy editorial"

I'd love to. Is there anyone willing to set it up?

@ Bruce

I can't afford the bandwidth on my colo box but I'll talk to the security group on campus and see if I can swing it.

@ quincunx

"it will inevitably come down to force (the state) or not (the market)"

So what's the distinction between being outvoted by your fellow citizens and being outvoted by your fellow shareholder(s)? ;-)

@False Data

Thanks for interesting commentary but I was a bit confused by this bit:

"You, and many of your readers, are passionate about security issues, but most of the rest of the world sees security, or the lack of it, as just one more thing in life they have to deal with."

It is true that many of the blog readers have strong opinions about security but I am sure they all deal with normal day to day life like anybody else. You can become habitually inquisitive and questioning about the percieved security of your daily normal life by practice and study but that is not the same thing as passion. I would argue that passion tends to distort good (security) judgement.

"... propose a mechanism by which society as a whole can make good security decisions even though many of the people in it might want less-than-optimal tradeoffs."

IMO, the only possible answer to a question like this education. Educating everybody in society to the level that they can deal with complex security problems does not seem practical or useful to me; that's why we have security specialists.

@ Grey Man

Thanks for the compliment. My guess is we're just disagreeing over terms. Here's the difference I'm talking about. I chose to label it as a passion about a subject, but I'm happy to use "interest" or some other word:

You and I read this blog regularly. I'd guess you have the same tendency I do to be willing to analyze security in terms of tradeoffs and security models, to follow security-related stories, and to be interested in the latest attacks.

Contrast that with most people I know who are much more interested in the latest episode of CSI, who's in the Superbowl, whether they got the tee time, and what time the kids' soccer practice starts. Their interest in security as a topic comes up when they're at the airport or getting their bag searched on the way into Sea World, or when they read the latest news about a suicide bomber in Iraq. They have absolutely zero interest in developing a security model or calculating the probability of an attack. As Bruce's essay says, as long as they feel pretty safe and don't have to spend too long at the airport security line, they're good to go.

I think there are a bunch more people in group 2 than in group 1, so I'm hoping Bruce will add something to his essay about how to deal with that fact. As you said, education might be a tough job, but if you go with experts instead, then you need to figure out how get from that expertise to a quality security policy that all the non-experts are at least willing to tolerate.

8th paragraph of the article:

"There is also direct research into the psychology of risk. Psychologists have studied risk perception, trying to figure out when we exaggerate risks and when they downplay them."

You should make your pronouns the same across the conjunction:

"...when we exaggerate risks and when we downplay them."

-or-

"...when they exaggerate risks and when they downplay them"

Upon reflection, it should definitely be:

"...when we exaggerate risks and when we downplay them."

Using "they" would imply that the psychologists are only investigating their own behaviors.

@ False Data

"So what's the distinction between being outvoted by your fellow citizens and being outvoted by your fellow shareholder(s)?"

Mu. The very framing of your question shows a great deal of misunderstanding of the market, which explains the initial call to mass democracy and state planning.

The most important difference is you are not bound to edicts of the charlatans voted in by your fellow voters, you can sell your stock and never buy again.

Now try to declare your property a sovereign territory not bound to any authority other than international law, and see what happens.

The ability to opt out is the ultimate check and balance.

" They have absolutely zero interest in developing a security model or calculating the probability of an attack."

Thank goodness. This way they can concentrate on their own comparative advantage to society.

---

@ Stephan Engberg

"d) I would REALLY like to see studies on why DEVELOPERS and IT buyers tend to underestimate the threat they are to others while they perhaps overestimate the threat of others to themselves."

I believe the analysis in the paper explains this under Prospect Theory.

They underestimate the threat to others because it is considered a loss (an externality), and therefore they will take their chances.

They perhaps overestimate the threat to themselves because it is a gain in knowledge and competitive advantage (internalization), and therefore they will keep it up.

Also, control is an issue that is also raised in the draft. An IT/Dev firm obviously has more control over itself than its clients.

"...or trading some security against a particular kind of explosive terrorism on airplanes against the expense and time to search every passenger..."

The string of "against" used with slightly different meanings is initially confusing. I would suggest something along the lines of:

"...or trading some security against a particular kind of explosive terrorism on airplanes versus the expense and time to search every passenger..."

"The more your perception diverges with reality in any of these five aspects..."

should probably be:

"The more your perception diverges from reality in any of these five aspects..."

"Why is it that, when food poisoning kills 5,000 people per year and 9/11 terrorists killed 2,973 people in only one year, are we spending tens of billions per year on terrorism defense and almost never think about food poisoning?"

It might help to drive home this point by using some actual numbers on the food-safety side:

"The Food and Drug Administration (FDA) performance budget request for FY 2007 is $1,947,282,000"

(From http://www.fda.gov/oc/oms/ofm/budget/2007/HTML/1PerformanceBudgetOverview.htm#Part)

"Like a squirrel whose predator-evasion techniques fail when confronted with a car..."

Good analogy. Another one that may be useful is armadillos: their instinct is to jump when approached by a car (which would otherwise probably pass harmlessly over them, if missed by the tires).

"Dealing with risk is one of the most important things a living creature has to deal with..."

The "Dealing...deal with" juxtaposition is a little awkward. Perhaps something like:

"Assessing and reacting to risk is one of the most important things a living creature has to deal with..."

"...the amygdala is what reacts immediately. It's what pumps adrenaline and other hormones..."

should probably be reworded...perhaps:

"...the amygdala is what reacts immediately. It's what causes adrenaline and other hormones to be pumped..."

Under "Risk Heuristics":

"How we get the risk wrong, and when we overestimate and when we estimate..."

is a bit confusing. Did you mean one of the following, instead?

"How we get the risk wrong, and when we overestimate and when we underestimate..."

-or-

"How we get the risk wrong, and when we overestimate and when we correctly estimate..."

"For half the subjects, the deck consisted of 70% happy faces and 30% frowning faces. Subjects faced with this deck were very accurate in guessing the face type: 68% of the trials. The other half was tested with a deck consisting of 30% happy faces and 70% frowning faces. These subjects were much less accurate with their guesses, only predicting a frown on 58% of the trials. The type of face affected accuracy."

This doesn't appear to be a congruent set of comparisons. Perhaps the second group wasn't

"...predicting a frown on 58%..."

but rather

"...predicting the face type correctly in 58%..."

"Groups of six observers watched a tw0-man conversation..."

typo in "tw0" should be "two"

"Linda is a 31 years old, single, outspoken, and very bright. She majored in philosophy As a student..."

The "As" should be "as".

"...and salient sensory input—but the issue really broader than that."

This should probably read:

"...and salient sensory input—but the issue is really broader than that."

"—and that I mean trade-offs that give us genuine security for a reasonable cost—"

probably should read:

"—and by that I mean trade-offs that give us genuine security for a reasonable cost—"

"Bruce, maybe you should consider vetting your articles on a wiki for easy editorial"

"I'd love to. Is there anyone willing to set it up?"

Hi Bruce, I'm a student at Polytechnic University's Information Security lab and we're all really big fans of your work. We discussed your need for a wiki and quickly agreed that we would be willing to set one up and maintain it for you. Send me an e-mail if you'd like to talk about this! :-)

-Dan Guido
dguido at gmail dot com
http://isis.poly.edu

You might want to change.... "If you misevaluate the trade-off, you won't accurately balance the costs and benefits."

Is misevaluate a word? Perhaps... "If you incorrectly evaluate the trade-off,...."

@ Bruce.

Another comment. You claim that there is an objective quantifiable risk, i.e. the likelihood of something occurring. But since the propensity of crime has a behavouristic component you easily end up in a circular argument. This likelihood is historic, but may not tell you anything about the future.

Example. Today people niavely assume that biometrics identification adds to security. But since biometris is inherentely spoofable and you cannot get new keys, there is in present models no fallback, no graceful degradation. So as soon as criminals and other forces (undercover agents, witness relocation and VIPs need new identities) get the tools and competences for systemic spoofing for Credential-based identity theft, the assumed "objective" risk change dramatically.

Security is dynamic unless you have context isolation in the system, i.e. damage control so you know as much as this is possible the worst consequence of (inevitable) security failure.

Without the dynamic component, you ignore that criminals learn - faster than new protections can be deployed.

@quincunx

Absolutely - but fact is that that is what is happening and a major factor explaining why we have so bad security. Almost all security is deployed to take control AWAY from you and thus reduce your security because someone else wants to have control.

Well, Trust (incl.) Privacy Enhancing technologies and for the above case a shift to on-card match of biometrics ONLY are examples that we can DESIGN risk down or away even in multi-stakeholder systems.

Risk reduction is the main design issue to deal with this dynamics - assume the server fails, assume the attackers succeed and then what !?

That the main reason why I advocate for a shift to National Id 2.0 where you incorporate Citizen Control through MANY identities or keys for logical separation of context. National Id is the present model only make risk and crime escalate.

I would suggest, in relation to the ordering of options part (linked to footnote 54), that at a certain point in list length, the emphasis will shift from the end of the list to the start of the list for a written question, but remain at the end of the list for a spoken question.
That would certainly appear to be the case in elections with long ballot papers for multiple candidates (standard Irish ballot paper would have 10 to 15 candidates, with at least 6 having a chance of getting elected, and there's a huge advantage to being at the top of the ballot paper - people get bored before they've read the whole list).

Forgot to say, though - great essay! Just kept getting curioser and curioser.

Hi. I also seem to remember some research that showed mens long term vs short term evaluations of risk are affected by the recent sight of a beautiful woman - if you see a beautiful woman, then your brain automatically decreases how much you worry about future ramifications. They did a test where you were offered an amount now or a larger amount in a months time.

Also, I'm interested in how all this feeds into gambling behaviour, since it is the most obvious form of people having an incorrect tradeoff strategy.

I don't think even 10000 years ago our responses were perfectly honed to our environment - superstitions have been around a long time, and are an example of feeling safer when you've done something with low to 0 effectiveness, and feeling less safe than you really are when someone has cursed you. I think, as long as we've been what we'd call human, we've had faulty risk assessment.

Page 11, 2nd paragraph: I think you meant "higher risk perception" rather than "higher risk perfection"

@ Food Poisoning comparison.

First, the question is not the size of the problem vs. cost, it is the size of the *mitigation* vs. cost. The achievable mitigation might be larger for food poisoning than terror attacks. Or it might not.

Second, security is like law enforcement and ticket punching. If done well, it seems superfluous.
So the question is not: Why are we spending more on security than attacks cost, but how much worse might the problem be if we reduced the investment?

(None of this is meant to undermine Bruce's points about where money is best spent, only the comparison with food poisoning, which does not respond to enforcement activity in the same way).

@kyberneticos:

I think that was the point of the evolutionary comparison. Our risk perceptions are not honed to protect us, as an individual. They are honed to protect our genes, which may or may not overlap. The problem today is that their is a bit of a lag between conditions and genome shift, in addition to our individualist assessment of value.

Additionally, I think that our culture get short shrift. Our culture is going to train us in ways that maximize the cultures survival and propagation, not our own. For example, many people today believe that they are immortal: that death is an illusion, which will be dispelled in the afterlife. Christians, Muslims, Hindus, New-Agers....

I doubt that this belief maximizes realistic threat assessment -- if I believe I'm immortal, instead of paying attention to the bear about to eat me, I'm going to be worried about how to properly pray. However, the culture of immortality is attractive. It can be easily evangelized. And so this belief spreads world-wide, in the face of all evidence.

Another example of the importance of culture can be seen in "Pigs for the Ancestors," Rappaport. In the New Guinea highlands, there has existed a cultural group that believed that pigs must be sacrificed to the ancestors to break a truce, and that that can only be done with adequate numbers of pigs, and the proper ceremony. This leads to truces enforced by the limited carrying capacity of the land.

When tribe A is preparing to attack their neighbors, it would be in the interest of their neighbors to mount a pre-emptive attack. However, if neighbor B has insufficient pigs, they'll hold off, believing that the attack will fail if they don't pacify the ancestors. Clearly a misguided assessment, in terms of the individuals of tribe B. But, in terms of the entire system, these biases in risk assessment limit war to the maximum supportable by the local carrying capacity, keeping the entire community of communities from collapsing, and so the belief lives on, century after century.

How many of our beliefs are similar? And could we identify them, if we wanted to?

@UNTER

Spot on. You're bringing up poli-sci as much as anthro.

Page 20, discussing the responses to the questions about divorce. This paragraph essentially says:

"In response to the first question, 23% said X; 36% said Y; 41% said Z.
"In response to the second question, 26% said X; 29% said Z; 46% said Y."

(Sorting the responses by percentage in each case.) I found this harder to compare the two cases than it would have been if the responses were in the same order in each case. That is, I feel it would be clearer as:

"First question: 26% said X; 36% said Y; 41% said Z.
"Second question: 23% said X; 46% said Y; 29% said Z."

It makes it easier to compare the two questions.

Bruce, I just got home from RSA. Caught your talk on Weds. My general comment about your talk--and admittedly I didn't get a chance to read your whole essay till now, so I am only basing this on what I heard live at RSA--is that you cited some very interesting things about human perception and our psychological foibles; however, I felt you failed to tie these in adequately to our thinking on security--specifically IT security, which is really what the conference was about.

My guess is that you ran out of time on the talk, right? The written essay seems to make better conclusions than what I heard in your presentation.

I really appreciate the thoroughness with which you research something when you decide to "go after it." That is a great quality. However, a professor of mine said of one of my essays that I was "mining nuggets" here and there, without adequately tying them into an overall framework of thinking.

The psychological studies you cite, while interesting, do not tell me why, for example:
1) Gen. Colin Powell, week after he retired as Secretary of State, was strip searched before boarding a plane for NY. Powell cited this in his closing RSA keynote address.
2) Furthermore, why Gen. Powell said he didn't mind the strip search in the name of security, even though admittedly, it added no security to the system.
3) Why, despite hard quantitative statistics, it is still difficult to justify ROI on a security budget.

Well, those were just some things I thought might be useful to address.

"I really appreciate the thoroughness with which you research something when you decide to 'go after it.' That is a great quality. However, a professor of mine said of one of my essays that I was 'mining nuggets' here and there, without adequately tying them into an overall framework of thinking.

Certainly my talk was guilty of that. I don't yet have an overall framework of thinking about this. Right now I have a lot of isolated facts and explanations, and I'm not really sure how they all tie together. I made a stab at it at the end of the talk -- and at the end of the essay -- but it's not really enough.

I'm still working on it, though. I don't know if this is a book-length idea, but it certainly is an idea that needs further research and analysis.

BTW, the way I think of it is this: any piece of writing needs both a "so" and a "so what?" Right now, the essay has a lot of "so," and not very much "so what?"

I'm working on the "so what?" part.

Small correction: an page number 14, in the seventh paragraph (starting with "Here's one...") in line two, in the word(s) "two-man", the letter 'o' seems to be of wrong size or font. Please correct this one in your OOo although it might only be noticable by very accurate readers ;-)

btw, *very* good essay so far. Can't wait until the whole thing is complete and published.

I am not a security person, nor do I play one on TV. I never finished my Anthropology degree. But I think a lot. My husband suggested I read this essay and contribute what I can. So here's a few thoughts:

1. The heart of your topic is philosophical, not psychological, in nature; Why do people react with fear to the unknown, and How do we differentiate between the New and the Dangerous. Psychology and neurology will offer insights to how the brain works, but they won't tell you everything.

2. Western political though has operated under the *incorrect* interpretation of Macchiaveli; he did not say it's better to be feared than loved, he said that the paranoid, double crossing, unethical Prince will probably live longer if the people are so terrified that they will not rise against him. He (Macchiaveli) says nothing about the health of the community or the people in it; the concept of "the people" as entities with rights was still a few hundred years away. When we rethink Macchiaveli, we'll come up with some more useful political philosophies.

3. Fight-or-flight is *not* the only response to stress. It is the typical MALE response. New research as reported by Psychology Review shows that there is a very different response typically seen in WOMEN: "tend and befriend." (also seen in "rally the troops", "circle the wagons," "Communicate and collaborate" "work as a team", etc. ) Read this review at http://www.apa.org/monitor/jan04/habit.html - And you will recognize this pattern once you think about it. A man who's had a crappy day at work goes home, pops a beer, and kicks the dog. A woman calls her girlfriends. Our current response to risk is the fight-or-flight method (hit back harder) and not the tend-and-befriend method (circle the wagons and get the attackers some therapy.)

4. Instead of looking for bigger, harder ways to hit back, maybe if we (meaning YOU because I know jack about security technology) were to consider tend-and-befriend methods to Risk assessment and management. One current method that comes to mind is certificates and certifying authorities. I trust the authority, the authority trusts you, so I can trust you. I'm sure you smart people can think of more ways to extend the "tend-and-befriend" metaphore to security.

5. Hackers in China and Russia wouldn't have to attack us if the societies in China and Russia provided sufficient opportunities for personal and professional growth. People who can earn an honest living are less motivated to steal. What if, instead of spending all our attention and money on stopping foreign bad guys, we just billed their government for the loss? Just a thought, like I said, I know jack about security.

Hi Bruce,
Your article is very similar to one written in a Time article (I think aroound Nov/Dec last year) about how bad humans are at evaluating risks.

What was very interesting in the article was a pyramid chart about how the various Americans who died last year (eg what did them in).

I dont remember numbers but basically pointing out aspects that more people died falling out of bed last year than shark attacks and similar parallels.

I mean lets face it Americans aren't afraid of eating hamburgers but more of them have been killed by that than terrorists.

Sir take off your shoes before you eat that big mac we want to do a cholestral check :)

Cheers,
Dean
www.Collins.net.pr/Blog

Bruce,

I really like where this is going but would like to see a clearly relationship between the front matter and the conclusion (as previously noted).

Also, is this sentence correct?

"Second, when considering security gains, people are more
likely to accept an incremental gain than a chance at a larger gain; but
when considering security losses, they're more likely to risk a larger
loss than accept a larger gain."

Should it not say "... accept a smaller loss."

Nice article.

A few comments:

1. Ethical consideration:

In my opinion, raising the perception of security to the real level of the risk is ethical, but raising it higher than the real level is not.

2. Reality/perception of the countermeasure

The reality/perception of a security risk is distinct from the reality/perception of an available countermeasure.

For example, there is a reality/perception of the security risk of getting a disease or attacked by a missile.

However, equally important there is a reality/perception people assign to the available countermeasures such as acupuncture or SDI.

3. Cost of a countermeasure?

Sometimes it’s is in negative dollars. Countering obesity by eating less, or lowering the risk of cancer by giving up smoking cigarettes saves money. Sometimes a countermeasure has no dollar cost - locking a door, or putting on a seat belt.

Two questions and thoughts on this very interesting and encompassing essay. My questions and thoughts center around the question whether some of the apparent errors of risk heuristics might not be errors at all when seen from a different point of view.

1. Risk from flying vs. risk from car driving: Certainly more people die due to car driving, but people spend more time with car driving, too. So something like deaths per km driven/flown or deaths per hour driven/flown might give a different view as to the comparable size of the risk

2. Natural vs. man-made risk: There is a significant difference between a deer ramming my car and a mugger stealing my money. The first is a basically random process and the probability of another deer ramming another car is not (too much) affected by previous similar events. This is different for mugging. A mugger who successfully acquired gains from his illicit trade will continue it, will even be an example for some other members of society, so the frequency of mugging might well increase. This might then lead to further risks becoming more frequent (see some suburbs for this). Therefore, if risk probability times damage is similar in both instances, it is very reasonable to fight the non-Markovian risk versus the entirely random one. The same goes, to a certain degree, for comparing deaths from terrorism to deaths from heart disease. The rate of heart disease will not increase dramatically if we do not fight it, the rate of terrorism might, if terrorists find that they can kill at leisure (to exaggerate)

Just these two ideas which might make some of the unreasonableness more reasonable.

Alexander

Under the heading Prospect Theory, please state that the subject was to invest $1,000. This can be inferred by the results, but it will allow the reader to make the connection faster.

You might find the sensemaking literature helpful. It focuses less on a single decision, and more on the dynamics of making sense of a specific context. I won't try to list my favorite articles/research, but here's a few starting points:

1. anything by Karl Weick - "The Social Psychology of Organizing" is my favorite, though most of his recent work focuses on high-reliability organizations

2. Gary Klein's Data/Frame model - from an area usually termed "naturalistic decision making"

3. Sensemaking research publicized by the Office of Force Transformation (www.oft.osd.mil) and the DoD Command and Control Research Program (dodccrp.org) - this research highlights the cognitive & social aspects of translating information into action (google for "sensemaking", "leedom", or "ntuen")

4. Although it's focused more on ontology than epistemology, I find Snowden & Kurtz's Cynefin framework useful.(http://www.research.ibm.com/journal/sj/423/kurtz.html)

5. Kevin Burns at Mitre has some interesting interactive sensemaking games/tools (mentalmodels.mitre.org) that allow people to more clearly see their biases.

Finally....it's telling that over 1/3 of the pages in the Spring 2006 MIT Press "Cognition, Brain, and Behavior" catalog were devoted to the category "Philosophy of Mind"....not a good indication that our knowledge in this area is anywhere near mature.

Great piece. I've only read half so far. My one comment is that I don't think it's fair to suggest that people are making an irrational decision in their fear of terrorist compared to food poisoning. Food poisoning kills more people every year, yes. But the food doesn't plot against the country. It is quite a different kind of threat. Risks involved in a terrorist attack, no matter how unprobable, when taken to their extreme could throw our country into turmoil for decades and threatens the fabric of our society. While it could be argued that things like the recent spinach contamination impacted the economy, that was mostly due to the overblown media coverage and doesn't compare to an agressive act by other human beings. Ecoli and botchulism don't have a cultures that clash with our own.

Bruce,

the latest essay is very compelling. I'm curious to know if you've considered speaking with Jeff Hawkins? In his book On Intelligence, his theory of intelligence posits that prediction is what the brain does. He describes an act of coming home and entering the house through the front door. You've done it so many times that without consciously thinking about it, your brain is predicting the experience like the sound the door makes, the weight of it as you push inward, the feel of the door handle, the tension of the lock as you turn the key. Then he suggests how the brain's predictions are knocked off course if he had arrived minutes before you and was able to significantly change the door's wieght. A change that you wouldn't detect upon initial approach to the door. Your brain just detects business as usual.

I think this is related to your notions of the security feeling and how we assess risk. Steve Johnson's brain must have picked up on something in the sound of the wind or the creaking wood or glass to trigger his need to move- a predicition. From as security perspective it would be great if we could know how this works and seek to emulate it with software. I think Jeff's company Numenta is on the right path compared to the failed AI attempts of the past.

I suspect the two of you could have a very good conversation.

And lastly,...you should write another book.

DC

I'm not an economist, but I'm pretty sure you're misrepresenting microeconomic theory when you quote the study about a sure gain/loss of $500 compared to a 50% gain/loss of $1000 in the "Prospect Theory" section. The fact that people chose differently when it's a gain, compared to when it's a loss, is not "unexplained" by economics.

When you said alternatives A and B have the same expected utility, you were making an assumption about the person's utility curve; namely, you were assuming it is linear. If you give me $20 instead of $10, I will be twice as happy.

Again, I am no expert, but I expect economists would assume a log curve, or other strictly convex function, is a better approximation of a utility curve. It certainly unlikely that it is linear, and there's a simple example.

Suppose my personal savings are $100,000 and you give me a bet where I have 50% odds of losing, in which case I owe you $100,000, and 50% odds of winning, in which case you give me $200,000. According to your version of utility theory, I would accept the bet because the expected outcome is in my favor. But very few people would accept this bet, because losing the bet and having $0 is much, much worse than winning the bet and having $300,000. To appropriately calculate the expected utility, you need to weigh the outcomes with the utility of that outcome.

It's the same reason we pay money for insurance. Insurance companies make a profit, so we know it's a bad bet. But paying out-of-pocket for a new car after a wrech decreases your utility more more than making smaller monthly payments.

Hence, we don't expect a gain and loss of the same amount to increase and decrease utility by the same amount. Of course, what economics doesn't know is what exact utility curve people have, but this is impossible to know. People have different curves, just as people have individual reactions to risk (which is also included in conventional micro-theory).

The "prospect" theory you mention seems out-of-place. The conclusions are logical, but my point is that I don't think they are anything new. Before mentioning it in a book, I would consult with a trusted micro-economist to clarify whether it really is a new theory to explain otherwise irrational behavior, or if it can be explained by conventional theory.

"The feeling and reality of security are certainly related to each other, but they're just as certainly not the same as each other. We'd probably be better off if we had two different words for them."

I would say that we already do: "Security", which relates to the mathematical/statistical side, and "Insecurity" (in its noun form), which relates to the emotional side.

I agree with Richard Braakman's analysis of the Linda example. I think it tells us more about the subjects' ability to comprehend lists of statements than their ability to estimate probabilities. Most of the statements listed do appear mutually exclusive, so it would be very easy to infer that "Linda is a bank teller" was also supposed to be a mutually exclusive option. The reader would thus interpret it as "Linda is a bank teller and not an active feminist."
This seems to be a moderately common fault in psychological tests: the psychologist assumes that the subject's understanding of the question is the same as the psychologist's own.

Great idea for a book Bruce. A chapter of interest to security folks would be a discussion of how the perception of risk (often due to litigation) impacts business decisions. Businesses will invest significant dollars because a jury would preceive a risk that, in reality, was minimal. Because the jury is convenced the risk was significant they'll award large judgements in cases where statistically it was not a "forseable risk" to the business. Just a thought.

Very good data about hueristics and trade-offs, although as a theist, I don't buy the evolutionary paradigm. Anyway, I would appreciate your adding how the research examples illustrate typical security scenarios.

I think class distinction plays into many people's attitude toward risks. In a suburban neighborhood with good schools where people argue about the neighbor's leaves blowing onto their lawns, many residents think they are ``above'' petty car vandalism, so when it happens, it's just that one guy in that one house that looks different from the other houses, must be him that's doing it. If only we could catch him in the act, it'd stop. Anyway, it's not the sort of thing that happens in _our_ neighborhood, so it's like it's not really happening. It's just an anomaly, and doesn't change the risk analysis. It's still so much safer here. It's just that one guy.

In NYC a lot of people learn to drive as adults, so they are quite bad at it and have strange ideas about what's appropriate. They are constantly breaking off mirrors, banging into bumpers while parallel parking, tailgating and stopping suddenly and causing 4-car pileups at 30mph. Trucks too wide for the street smash into parked cars. And this seems to cost more to fix and happen many times more often than the guy who smashed my friend's window to grab his CD's. It's my problem to worry about it, so I worry about the hit-and-run trucks and the horrible drivers. but ask an outsider from the other side of the pseudo-class divide, ``OMG you took your suburban car to The City? Wow, I hope you parked in a garage. It's not _safe_ there---those people, they're not like us. They'll just take a baseball bat and smash every car window on the block, because they're angry at you for having a car, that's why they do it!'' The wildly-inflated risk analysis is really just disguised class arrogance. It has much more to do with pride than fear.

It works the other way, too. In Manila, bombs go off in the mall or on the elevated train every year or two, and people die, and people get angry, maybe even call the bombers ``animals,'' whatever, but since most don't consider themselves ``too good'' to put up with this risk, it's analyzed next to murder/rape/kidnapping/traffic-accident, and the mall and the train remain crowded, relatively happy places. But if a bomb goes off in the U.S. (as long as it wasn't set by white supremacists), the fucking sky is falling and ``I thought stuff like that only happened in third-world countries, not here in the Greatest Nation [blah blah]. Suddenly we're not invulnerable!'' and officeladies in sneakers are weeping loudly in the streets. I think their pride was injured as much as their feeling of safety.

@Ben Liddicott:
"The lottery is not an investment -- it is entertainment, a fee for the pleasure of sitting around discussing what you will do if you win. People pay a dollar into their workplace pool so they can then chat about it, not because they think it is a good investment."

Here's another way of looking at paying into your workplace's lottery pool: insurance. If all your colleagues became millionaires and you didn't, not only would it be emotionally painful, you'd end up having to do all their work after they quit as well.

Of course the smarter approach would probably be to get a real insurance company to insure you against that risk. The UK national lottery pays out something like 50% in prizes. Your insurance company might only be making around a 10% margin on premiums though, meaning they pay out 90% in "prizes" - so the insurance premium ought to be cheaper than buying the lottery ticket.

When you say "9/11 terrorists killed 2,973 people in one non-repeated incident", I think the reason we spend billions on it is because we believe it *would* be repeated otherwise. Some of us think something like it will anyway, even with the "security" that's been put in place. The cost of security is based on the perception of *future* risk. For ongoing threats, statistics are a valid approach, but since the 9/11 event changed that particular game calling it "one non-repeated incident" underplays the threat.