Schneier on Security

Bruce, can I use you as a reference when my wife looks at me strangely when I voice simlar thoughts?

I think this way all the time. Hotels do a similar thing a lot: they ask you to leave your key at the reception, then give it, it the worst case, based on the room number alone. And the cleaning ladies often with hold a room door open for you if you approach it confidently.

Interesting that you've just posted this, especially since I've just read another article on the perceived parallels between the thinking processes required in security and the thinking processes required in formal mathematical analysis. You might like to have a look at it, at http://www.daemonology.net/blog/2008-03-21-security-is-mathematics.html .

Is it just security professionals, or can this apply to aspects of the beta-test/programmer community?

What little coding I did when working on an online game, I always beta tested from the perspective of 'how can I break this?' or 'how can I exploit this?'

As for the 'hotel cleaning ladies holding open the door,' that's simply deferring to authority. Unless they've seen you come out of another room, they really have no means of verifying your identity, and are responding to your manner. This plays out in other places, as well - you're likely to be the one people approach in a store, no matter how you're dressed, and asked, "Do you work here?" Or, in a crowded environment, people part to let you pass, because you are moving with intent. (That last drives my wife nuts, since she has to scramble to keep up with me - any gap closes behind me.)

@David

"Bruce, can I use you as a reference when my wife looks at me strangely when I voice simlar thoughts?"

I've tried this and it doesn't work. Being security conscious means being a criminal, if only in one's head, and this spooks people.

While not directly related to security, I would love to see this mindset applied to legislation.

All bills are just absolutely wonderful according to proponents. The question that is ignored is "how will this bill be misused?"

Case in point: http://www.boingboing.net/2008/03/25/fake-craigslist-ever.html

"Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail."

-I tend to disagree; good engineering requires one to think about various failures (intentional or non-intentional), and how to cope with those failures.

@Bruce .. it explains some of your diatribe against things that "normal" people won't think twice about. And frankly I will never think about sending a a tube of ants to someone I care or don't care about.

There is always Zprexa .. used to be that Prozac did the trick .. but the world has gotten to be a lot nuttier.

@John:

Is the tradeoff of such legislative design a restrictive environment where individuals are not trusted to decide or accept responsibility? I'm not sure that laws which assume you're not smart enough or honest enough to comply are an improvement.

Fred P is correct. Petroski had a book about this (forget the title) that said engineering is about foreseeing failures in the engineering process.

So the premise fails. Everytime I hear the wornout line of "ooooh security guys and hackers see the worlds weaknesses" I think that is the equivalent of security-porn. Heaps of people in different jobs think this way, engineers, police officers to catch crooks, intelligence analysts, accountants doing your tax. Each of them think about failures in their particular domain, and probably take those failure-perceptions into other areas.

So please, the only reason the security guy/hacker world view is continued on blogs like this and others is to make you guys think that you are special and unique. You arn't.

@YWo: I don't know where you hang out, but most of the people I know do NOT think this way. If engineers, police officers, intelligence analysts, and tax accountants really did think this way, the world would be a very different place. Who would the TV reporter interview at the airport security check if everybody was in on the joke?

RE: Hotels

I've stayed at quite a few hotels and have had different experiences, when I walked in my room when the cleaning lady was there (door was open) she asked for my room key and made sure it worked in the door. This hotel also required your roomkey to use the elevators.

At another hotel when I've lost my roomkey, a photo ID was required to retrieve it. These were upper class chain hotels acustom to business travellers tho, not sure how it would be for other hotels.

There is the human factor too. We are social animals, we respond to cooperation, and, at least within our own group, do not want to be constantly 'on guard', thinking the worst. We seek out environments and people where we can trust, can let our guard down. (how often have you heard people yearning for a place where you 'don't need to lock your doors')

It starts to come down to tradeoffs. How much risk are you willing to take. Do you really want steel bars on your home? I don't always lock my car (hell during the summer it has neither roof nor windows), thought I will usually pop off the radio face plate and lock that up. To view every possible security threat at every possible time would result in paranoid, crippling life style.

My very first experience programming, well aside from toggling in the boot sequence on a PDP 8L (IIRC), was discovering that I could alter the BASIC code of the Lunar Lander program on the mainframe shared by all the schools in my state. Remember Lunar Lander?

M--------L

The first thing I discovered was I could change the lander symbols

)----->

Then I discovered I could alter the If-then statements for landing conditions, and (being 15) I quickly coded all sorts of horrific disasters as consequence for different landing velocities.

The 110-baud dial-up modem connection I was using kept disconnecting, which inspired me to add a randomly accessed subroutine that simulated a disconnection, then simulated a login sequence, and wrote the entered usernames and passwords into a file. After that the program ended, dropping you at the OS prompt where you would expect to be after login.

I collected a LOT of usernames and passwords...

Then there was the time that I worked for an energy-management firm, responsible for heating and cooling public and private buildings. I noticed that anyone could phone the 300-baud modem connected to the mainframe and immediately connect, with no password. I pointed out to my boss that one could disable the alarms on a school boiler, shut off the pressure valves, and then start the boiler. Someone could blow up a school!

I nagged and nagged that the modem should be kept turned off, and if a building engineer wanted to use it that they would phone in, identify themselves, and ask for the modem to be activated. Nothing happened.

Then the movie "War Games" came out, with its plot about kids accidentally dialing into a mainframe and starting a nuclear war. Suddenly my boss developed a new security procedure wherein the modem was kept shut off...

@YWo who said:

"Fred P is correct. Petroski had a book about this (forget the title) that said engineering is about foreseeing failures in the engineering process. So the premise fails."

My comment:

you obviously lack the skill to read between the lines, which proves Schneier's whole point he was making.

When engineers look at weaknesses and failure they look at the consequences of a failure, not how to make the system fail.

In many offshore systems a "safe fail" is often desired over a "fail safe" system. In other words, a system that fails without consequences.

The security mindset goes in the way of seeing how you can make the system fail.

BTW: I see many ways of having airport security fail, specially in the US. First you take away all weapons by security check, than place armed people on the other side. Overman one, and you have a weapon.

Great article Bruce.

I am reminded of some of the flim flam that James "the Amazing" Randi exposes. Even more than the exposes, what is really impressive is how he sets up the experiments to avoid security breaches. Conditions that seem absolutely watertight at first glance turn out upon inspection to have damning vulnerabilities which are invariably the vector for the Uri Gellers of the world to exploit. Randi finds them, plugs the leaks, and poof, the paranormal abilities disappear. It takes a very special mindset to think like that. I have always envied it, and despite a reseach education that actively encourages questioning your results, don't quite have it.

May I ask a question here that may be relevant? It has always puzzled me. Mr. Schneier, how do you avoid spambots in the comments?

The book was "To Engineer is Human", by Petroski. Required reading my freshman year.

Agree totally with Fred P. GOOD engineers (and designers of all stripes) pay attention to all sorts of failure modes, whether security or not.

Plenty of bad or lax engineering abounds, however. And now that I think back to e-school, it seems that maybe they need to add a class on critical design thinking, and predicting failures before they happen.

The "security mindset" and the "criminal mindset" have some obvious similarities. What do you say to those people who are inclined to take those similarities as being deeper and more essential than they actually are?

Fred P. is not totally correct. Bruce has said it many times you want to see how things fail and as a designer you want failures to be predictable and of minimal impact. You don't want large catastrophic failures and knowing how it fails helps with code execution and overflow behavior I would suspect. Sure you can try and design it to never fail, but designing to fail in predictable safe ways is far more realistic and advantageous. The way the twin towers went down is a perfect example, they took the heat for approximately 2 hours as designed and then collapsed in a way that probably saved as many lives as it cost because of the design by the engineer originally.

@sooth sayer

Some people think about sending ants, other people actually do it. I'd like the thinkers to think about it before the doers get a chance to do it.

Engineers have got to consider failure to be any good. For example, a lazy engineer might specify the strongest bolt of a given size to hold together two halves of an assembly. But if the two halves are complicated, expensive castings, perhaps you'd rather have the bolt break under stress than a corner of the casting.

Such considerations are much more straightforward though than those surrounding most security situations where inexact sciences like economics and psychology come into play.

@john "The question that is ignored is "how will this bill be misused?""

I think you will find the related question "How can I be sure I can misuse this bill" is uppermost in the minds of many legislators.

"The 'security mindset' and the 'criminal mindset' have some obvious similarities. What do you say to those people who are inclined to take those similarities as being deeper and more essential than they actually are?"

I think the similarities are pretty deep and essential. The difference is how you act on it: making things more secure vs exploiting the insecurities for personal gain.

I must have that mindset. the first time I read about FACS ( facial action coding system) I immediatly realized that botox would eliminate its effectiveness.
There are now computer facial recognition systems that read FACS.

The power grid would be more secure?

Maybe if they could find a safe and inexpensive way to bury transmission lines and substations a hundred feet down.

@Nick Lancaster:

Yes, absolutely! In an ideal world, every programmer would be trained in the security mindset.

It also helps because there's a lot of overlap in security and usability problems. Usabilty issues also often spring from never questioning the assumption that people will only use a program in a certain way-- for security, the assumption is you'll only get authorized users, and for usability, it's that you'll only have expert users who instinctively understand the programmers' mental model of the program.

"How could somebody inadvertently do something they didn't want to?" is nearly equivalent to "How could someone intentionally break this?" for many programs, and the first question may be received better than the suggestion that your users are criminals.

@Nick
When people refer to laws being abused, they are generally referring to laws that give more power to government officials.

The Jason Bourne books by Robert Ludlum contain much of this pattern of thinking. I recommend them to anyone in this discussion who has not yet read them.

@ vvpete

> Engineers have got to consider failure to be any good.

Certainly. However, for most engineering applications, failure modes examined are unintelligent failure modes. What happens if the lateral force on this building design exceeds N? What happens if the soil density under this building is affected by a flood? If lightning strikes this power line, how much damage will the surge do to whatever is connected to it?

Security professionals, on the other hand, consider a different type of failure: the intelligent failure or the engineered failure - failure modes that aren't due to natural occurrences (or even freak occurrences), but the intentional gaming of the system by an intelligent entity.

Engineering is close to security... in fact, in practical terms often times it's better to listen to the engineer than the security guy because those natural or freak occurrences are often times orders of magnitude more likely than malicious misuse.

@ Dave Walker

re: Percival's blog

That's an interesting post. I don't know that I buy it, entirely. Mathematics can certainly teach you rigor, but mathematical systems are usually closed, and real world security systems are usually not. I do think that many mathematicians can be good security guys, but I'm reminded of one of my favorite of Bruce's anecdotes from his Applied Cryptography days. He's at a conference, and involved in some discussion when an FBI (?) guy is describing a side-channel attack, and Bruce says, "But that's cheating", and the lightbulb comes on.

Classically trained mathematicians can easily wind up being stuck in a box. Axiomatic systems can do that to you. :)

>>"The 'security mindset' and the 'criminal mindset' have some obvious similarities. What do you say to those people who are inclined to take those similarities as being deeper and more essential than they actually are?"

>I think the similarities are pretty deep and essential. The difference is how you act on it: making things more secure vs exploiting the insecurities for personal gain.

Two non-technical writers I can think of have used "criminal mind" in the sense of "security mindset", and clearly understood its usefulness.

Roald Dahl, in one of his short stories, Parson's Pleasure, talks about the local petty-crook farm boys taking great interest when an antique dealer purports to show them that their piece of furniture is a modern replica, because it has machine-made screws. They are fooled (he has palmed the actual and very old screw), but they clearly have the right idea.

And more recently, Anthony Bourdain, in The Nasty Bits, uses the phrase in reference to his restaurant line cooks, who he seems to think need this attribute to be great at what they do.

@pohart

So are we better served by having unethical thugs restricted by carefully-worded laws, or by elected representatives who can be trusted to follow the spirit of the law? Would we even want public servants who *need* to be limited in such a manner (yes, I understand we've got more than a few people in D.C. playing the 'that which is not explicitly forbidden is legal' game)?

Can overly restrictive laws also work against the populace, in the sense that a healthcare law, focused on making sure hospitals and employers aren't defrauded, end up burning the citizen?

I'm a programmer, so I build things for a living. I founded my own company, so I wind up in charge of security and building things at the same time.

One thing I've noticed is that I can't do them on the same day. Either I'm thinking about how to make things work and building them, or I'm thinking about how things are exploited and breaking them. Trying to do both gives me a headache and never works.

I think the hardest thing to deal with is the reaction of people who do not think this way. I find that they sometimes react negatively to my analysis ("only a criminal would think like that") as opposed to reacting negatively to the crummy security.

This really relates to Bruce's previous post of Adam Shostacks's Security Development Lifecycle blog, which I found to be an excellent resource as backing for process improvement where I work. There are people who are good at breaking things. There are people who are good at creating things. Sometimes traits co-exist in those extremely talented individuals you come across. That definitely hasn't been my experience. I've been carrying the 1-ton "paranoid" gorilla on my back for many years now. (My posture is pretty bad because of it- can't blame age for everything.) The developement process has to include us paranoid types to make it work. Judging the continuing state of infosec, this hasn't been accomplished. I'm confident that the pieces are coming together though, especially seeing posts like Mr. Shostack's.

I've noticed several elements to the security mindset, both here and in previous posts.

1. A tendency to notice the way things might fail and how someone might exploit those failures.

2. A tendency to point out or explain those security flaws.

3. A tendency to place a stronger value than most people place on addressing security flaws, such as addressing flaws that others might regard as uneconomical to fix or as economic externalities.

Are all three necessary for a security mindset? Is any one of them sufficient?

This mindset is really no different from that of people who work in areas such as product safety, aviation safety, construction safety, fire prevention, etc.

i grew up around engineers who thought this way. they mostly spent their time doing failure analysis, so i guess it makes sense.

it's eminently disturbing that this sort of thinking isn't more common. people take too much for granted. seems a foundational element of critical thinking- "when will this idea _not_ work as intended? is that acceptable?"

@ Petrea
""How could somebody inadvertently do something they didn't want to?" is nearly equivalent to "How could someone intentionally break this?" "

Hmm, I'm not convinced. While both seem to cover the immediate issue (domain) failure, the second contains a factor not in the first, and that is intent. Implied in that is a reasonable assumption (ok I'm a security mindset sort) that the intent would be followed by further explaoitation, and that requires a security person to drive solutions deeper and provide more systemic, robust answers than just the immediate issue.

They are certainly close, but the quantum leap from "every job looks for failures" and "Security Mindset" is the extension from the immediate domain to the overall system IMO.

You can send someone 1500 live ladybugs for $10 via Amazon:

http://www.amazon.com/1500-Live-LadyBugs-GOOD-BUGS/dp/B000MR6WRG/ref=pd_bbs_sr_2

It doesn't take much imagination to think of the practical jokes you could play when armed with $1000 worth of ladybugs.

Lancaster:

"Is the tradeoff of such legislative design a restrictive environment where individuals are not trusted to decide or accept responsibility? I'm not sure that laws which assume you're not smart enough or honest enough to comply are an improvement."

It looks like we aren't talking about the same thing.

I'm not talking about the citizen exploiting a law. I'm talking about officials exploiting laws that were apparently well intended.

Especially in criminal law, it is extremely important to consider the potential for official misuse.

The link to the 'smart pillbox' security review at UW is broken, here's the correct link:
http://cubist.cs.washington.edu/Security/2008/02/10/security-review-smart-pillboxes-maybe-too-smart/

Westlake is a mystery writer. He writes about characters that think like that.
Shouldn't cops think like that? Except I never read police procedural novels that have characters that think about security.

I also have to object about the engineers not thinking about how to make things fail. How else does the term 'social engineering' come about? You find out how something works or doesn't work. Then you use that to your end for good or evil.

I think Colin Percival's blog post overstates the connection between mathematics and the "security mindset" by implying, implicitly, that all the security mindset takes is rigorous attention to detail. As Pat says, attention to details, in and of itself, does not prevent you from missing the big picture.

However, if we limit ourselves to mathematical research (and not merely passing classroom courses) a much better argument for the proposition comes from the observation that the best way to devise proofs of interesting theorems is often to try to construct a counterexample.

Once a proof is arrived at, mere rigor is all that is needed to check it. But finding it in the first place (and, just as important, finding out what to prove!) often demands a more creative outlook.

You start by hypothesizing that conditions A, B, and C are sufficient to guarantee the desired result R. Right after this you mentally switch to being "the adversary" and try to figure out a way to achieve not-R while still satisfying A, B, and C. If you're in luck, you fail to find any counterexample, but do manage to convince yourself that your search for counterexamples was exhaustive. The narrative of your search then becomes the first draft proof, which if you're a good mathematician you will probably be able to massage into something more direct and elegant.

Another way to be in luck is to actually find a counterexample. The counterexample shows that your hypothesis was not good enough; you system could be broken. But in most cases the counterexample will also point towards additional assumptions you need to add to your conjecture. So you still achieve progress.

The only way not to make progress is not to find any counterexamples, and also not to be sure that one looked closely enough for them.

Thus, in order to be a successful mathematician, one needs the ability to look at some complex system (the assumptions) and try to find a way to beat it - probably by guesswork and hunches at first (because that is quicker), but hard and systematic if that does not work. This appears to me to be very close to the "security mindset" that Schneier describes.

I acknowledge that not all mathematics is done in this way - it depends on the subdiscipline how often it pays off - but in my experience a lot of it is.

I'm afraid I can't find a link, but I remember a car being stolen in the Bay Area in just the fashion you describe (I believe it was a Lambourghini from Stanford European). Basically the thief walked in, said he was there to pick up 'his' new car, and walked out with the keys. Apparently the car was last seen heading south on 101 towards Los Angeles, so the assumption in the news article I read was that the car was destined for Mexico.

"You can send someone 1500 live ladybugs for $10 via Amazon."

Good to know.

But the ants are completely anonymous; the ladybugs require you to leave a credit card number.

"@Bruce .. it explains some of your diatribe against things that 'normal' people won't think twice about. And frankly I will never think about sending a a tube of ants to someone I care or don't care about."

Of course you won't. Most people won't. The point is that there are bad guys who will try to abuse any system out there, and a good security engineer needs to be one step ahead of those bad guys.

@ Nick Lancaster

"(yes, I understand we've got more than a few people in D.C. playing the 'that which is not explicitly forbidden is legal' game)"

When was this ever confined to people in D.C.?

I've not gotten jobs a few times now because the interviewer asked questions along the lines of: "Have you ever thought about how you would steal from a store?" Well, duh, that was part of my job at a bunch of places, so as to work out how to discourage theft. They don't want to hear that when you're going to be handling their money.

I would go so far as to say that most people don't want to think that the people around them are working out where the chinks are in the armor.

What's depressing about this enlightened thread is that the whole human effort to secure things, or even to invent things, becomes so futile.
Everything has a weakness waiting to be exploited, a goal, however good, to be thwarted or twisted. It's as if humankind would be better off simplifying life, understanding the basics of human survival and satisfaction, to improve existence. But, to do that will probably require global cooperation and equanimity-- peace. The whole game of outsmarting the other guy is a doomed cycle of endless frustration--the big picture would be to eliminate the need for security. Not by having big brother draconian surveillance, but by making it possible for everyone to meet their needs without conflict. If a human can think it, it might be possible. Cynicism and observation sometimes lead back to simple solutions, even if they appear naively idealistic.

Ronald van den Heetkamp how does it prove his point and what exactly am I reading between the lines for?

You are an idiot who cannot construct an argument correctly because you are ambiguous in your claims and you don't state how it proves anything.