Schneier on Security
A blog covering security and security technology.
« GPS Spoofers |
| Friday Squid Blogging: Squid Vision »
March 2, 2012
Liars and Outliers: The Big Idea
My big idea is a big question. Every cooperative system contains parasites. How do we ensure that society's parasites don't destroy society's systems?
It's all about trust, really. Not the intimate trust we have in our close friends and relatives, but the more impersonal trust we have in the various people and systems we interact with in society. I trust airline pilots, hotel clerks, ATMs, restaurant kitchens, and the company that built the computer I'm writing this short essay on. I trust that they have acted and will act in the ways I expect them to. This type of trust is more a matter of consistency or predictability than of intimacy.
Of course, all of these systems contain parasites. Most people are naturally trustworthy, but some are not. There are hotel clerks who will steal your credit card information. There are ATMs that have been hacked by criminals. Some restaurant kitchens serve tainted food. There was even an airline pilot who deliberately crashed his Boeing 767 into the Atlantic Ocean in 1999.
My central metaphor is the Prisoner's Dilemma, which nicely exposes the tension between group interest and self-interest. And the dilemma even gives us a terminology to use: cooperators act in the group interest, and defectors act in their own selfish interest, to the detriment of the group. Too many defectors, and everyone suffers -- often catastrophically.
The Prisoner's Dilemma is not only useful in describing the problem, but also serves as a way to organize solutions. We humans have developed four basic mechanisms for ways to limit defectors: what I call societal pressure. We use morals, reputation, laws, and security systems. It's all coercion, really, although we don't call it that. I'll spare you the details; it would require a book to explain. And it did.
This book marks another chapter in my career's endless series of generalizations. From mathematical security -- cryptography -- to computer and network security; from there to security technology in general; then to the economics of security and the psychology of security; and now to -- I suppose -- the sociology of security. The more I try to understand how security works, the more of the world I need to encompass within my model.
When I started out writing this book, I thought I'd be talking a lot about the global financial crisis of 2008. It's an excellent example of group interest vs. self-interest, and how a small minority of parasites almost destroyed the planet's financial system. I even had a great quote by former Federal Reserve Chairman Alan Greenspan, where he admitted a "flaw" in his worldview. The exchange, which took place when he was being questioned by Congressman Henry Waxman at a 2008 Congressional hearing, was once the opening paragraphs of my book. I called the defectors "the dishonest minority," which was my original title.
That unifying example eventually faded into the background, to be replaced by a lot of separate examples. I talk about overfishing, childhood immunizations, paying taxes, voting, stealing, airplane security, gay marriage, and a whole lot of other things. I dumped the phrase "dishonest minority" entirely, partly because I didn't need it and partly because a vocal few early readers were reading it not as "the small percentage of us that are dishonest" but as "the minority group that is dishonest" -- not at all the meaning I was trying to convey.
I didn't even realize I was talking about trust until most of the way through. It was a couple of early readers who -- coincidentally, on the same day -- told me my book wasn't about security, it was about trust. More specifically, it was about how different societal pressures, security included, induce trust. This interplay between cooperators and defectors, trust and security, compliance and coercion, affects everything having to do with people.
In the book, I wander through a dizzying array of academic disciplines: experimental psychology, evolutionary psychology, sociology, economics, behavioral economics, evolutionary biology, neuroscience, game theory, systems dynamics, anthropology, archeology, history, political science, law, philosophy, theology, cognitive science, and computer security. It sometimes felt as if I were blundering through a university, kicking down doors and demanding answers. "You anthropologists: what can you tell me about early human transgressions and punishments?" "Okay neuroscientists, what's the brain chemistry of cooperation? And you evolutionary psychologists, how can you explain that?" "Hey philosophers, what have you got?" I downloaded thousands -- literally -- of academic papers. In pre-Internet days I would have had to move into an academic library.
What's really interesting to me is what this all means for the future. We've never been able to eliminate defections. No matter how much societal pressure we bring to bear, we can't bring the murder rate in society to zero. We'll never see the end of bad corporate behavior, or embezzlement, or rude people who make cell phone calls in movie theaters. That's fine, but it starts getting interesting when technology makes each individual defection more dangerous. That is, fisherman will survive even if a few of them defect and overfish -- until defectors can deploy driftnets and single-handedly collapse the fishing stock. The occasional terrorist with a machine gun isn't a problem for society in the overall scheme of things; but a terrorist with a nuclear weapon could be.
Also -- and this is the final kicker -- not all defectors are bad. If you think about the notions of cooperating and defecting, they're defined in terms of the societal norm. Cooperators are people who follow the formal or informal rules of society. Defectors are people who, for whatever reason, break the rules. That definition says nothing about the absolute morality of the society or its rules. When society is in the wrong, it's defectors who are in the vanguard for change. So it was defectors who helped escaped slaves in the antebellum American South. It's defectors who are agitating to overthrow repressive regimes in the Middle East. And it's defectors who are fueling the Occupy Wall Street movement. Without defectors, society stagnates.
We simultaneously need more societal pressure to deal with the effects of technology, and less societal pressure to ensure an open, free, and evolving society. This is our big challenge for the coming decade.
This essay originally appeared on John Scalzi's blog, Whatever.
Posted on March 2, 2012 at 1:21 PM
• 43 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
I was already quite aware of the book, and I have the highest esteem already for you Bruce, but I wasn't going to read it because, well after all, I've been reading your stuff for over a decade.
But this small essay is so much more interesting than I had imagined the book to be, that it prompted me to immediately suck the kindle e-book down and now I'm reading it on my nook.
There's an interesting step in the there, somewhere, directly related to the subject at hand, too.
I think I might have inadvertently left an ambiguity in my comment: just to be clear, I very happily paid for the kindle edition on Amazon.
Can that claim ("not all defectors are bad") be generalized to all types of network or connection, of any kind? Whether we're talking about the network of genes in our chromosomes, a communicating connection of slime molds, or a society of humans, stasis is the ultimate killer.
In other words, perhaps it is feasible to posit that human society could do more than merely accept that defectors exist; a wider definition of society could include those defectors, so that the rest of society can learn whatever new knowledge the defectors discover, either to benefit from or protect against. An example, in a subset of the human network, might be the hiring of hackers to expose weaknesses in a computer network.
Many moons ago 9/11 happened and in the ensuing conversations you enabled I made the point that "one man's terrorist is another man's freedom fighter" it is at the end of the day a question of the viewpoint and thus the morals of the audience not the actor.
I have also pointed out that the easiest way to lie is to tell the truth but from a different viewpoint. This puts a twist into the other persons thinking and due to generalised human cooperation brings them around to that viewpoint. In essence debating is about viewpoint changing.
Society like life is a journey we learn as we travel, what may once have been acceptable for pragmatic reasons lingers after the reason has gone, and society finaly catches up.
In your book you give some graphs most of which are when viewed from a distance first or second order response curves to either step or impulse inputs. Which is what you would expect, the more conservative viewpoint producing a more damped response.
Whilst individuals and even small groups do not of necesity follow this response, the overall response of large societies tends to. This is a little like "brownian motion" individual particals go in all directions but overall the higher energy particals rise against a gravity gradient, and also give up energy to other particals in their progress, resulting eventualy in a near even distribution in a closed system.
All of which gives rise to a question that many have asked in the past "can control theory be used to predict society?". In the past it has always been left as an open question, due to insufficient computing power. However we do know from economics that at times of slow to moderate change, even simple mechanical models will make sensible predictions.
How ever significant money is not made at times of moderate change, it is only made in significant quantities at times of high volatility be it visable or invisable to the general market. Which is why we had the banking crisis, defectors were activly creating instability in order to maximise their returns. Part of this was by hiding information from market models so that the market response was over damped to the changing but hidden inputs.
And this is a problem that is going to be dificult to detect which is "hidden defectors" who publicaly go with society whilst secretly acting or betting against society. We have names for them such as "a wolf in sheeps clothing"...
The question thus arises is do we have enough computational resources to build control theory models of society that are either accurate, or perhaps more importantly sufficient to unmask hidden defectors?
@Oliver "a wider definition of society could include those defectors, so that the rest of society can learn whatever new knowledge the defectors discover, either to benefit from or protect against."
I think Bruce alludes to this in his book. The defectors, how ever you want to think of them, are the innovators. They are the ones that challenged the status quo for good or for bad. Whether it was Martin Luther King (good) proposing a radical concept of social equality during the civil rights era or on the flip side Adolf Hitler (bad) who sought to unite the Germans after World War I .
Suppose you could theoretically remove the defectors, then you would have conformity. There is a reason why the most innovative nation is also a democracy and also why you don't generally see a lot of innovation going on in dictatorship countries.
While true that society needs deviants in order to advance, one should also be aware of what Francis Bacon wisely wrote so very long ago: "A froward retention of custom, is as turbulent a thing as an innovation".
In context, that can be just as disruptive as defection.
@Clive "Hidden defectors"
I like that. Never really thought about it that way. The Banking crisis example you gave was a nice illustration.
I think however, that it isn't so much that they necessarily hide their defection(although they can), their actions just are not observable. More importantly though, the impact of their actions are not observable either.
However, there comes a "tipping point" when all of a sudden critical mass is reached. Now the "hidden defectors" collective actions are observable.
In a situation like the civil rights movement, you have a famous defector like Martin Luther King who pushes the system over that tipping point. However, the hidden defectors (the people who thought inequality was wrong but didn't publicly say it) are the ones that made it possible for King to have such an impact.
Could someone comment about the extent to which Bruce's politics permeates the new book?
The Gospel of Rand teaches us that if everybody acts in a purely selfish manner, a perfect equilibrium will be achieved.
@DB Cooper "In context, that can be just as disruptive as defection."
You need disruption for advancement. Therefore you need continual disruption for continual advancement.
There is a reason why there is a status quo. It is because people are comfortable where they are at and many fight to preserve it because change will hurt them. The technology sector is flush with examples. (Kodak,Palm, Rim, ......)
Are there absolute lines that a defector can not cross? I'm thinking of Capt'n Kirk and Star Trek.
You have the prime directive, but he still interferes when the society is breaking absolute rules like, you don't put people into suicide machines, or don't let a computer run society.
Unfortunately who is an outlier and who is dangerous is a very subjective call. Is possession of a lock pick a crime, or a tool? What about stuxnet source code?
@Frank Ch. Eigler: I don't think it is possible to think honestly about these questions without drawing political conclusions from the answers one reaches, whatever they are.
Presumably the book now permeates Bruce's politics.
So what will happen if you keep generalizing past "the sociology of security"?
I'm guessing the next logical step would be to move on to the biology of security. Then on to the chemistry... and then the physics... and then all the way to... uh... the mathematics.
I don't know if you've ever read 'the abolition of man', but it bears on the subject. You list onlybexternal pressures as if man is a marble being pushed. Yet morals come from within
Let us suppose for a moment that the harder virtues could really be theoretically justified with no appeal to objective value. It still remains true that no justification of virtue will enable a man to be virtuous. Without the aid of trained emotions the intellect is powerless against the animal organism. I had sooner play cards against a man who was quite sceptical about ethics, but bred to believe that 'a gentleman does not cheat', than against an irreproachable moral philosopher who had been brought up among sharpers. In battle it is not syllogisms that will keep the reluctant nerves and muscles to their post in the third hour of the bombardment. The crudest sentimentalism (such as Gaius and Titius would wince at) about a flag or a country or a regiment will be of more use. We were told it all long ago by Plato. As the king governs by his executive, so Reason in man must rule the mere appetites by means of the 'spirited element'.
And all the time—such is the tragi-comedy of our situation—we continue to clamour for those very qualities we are rendering impossible. You can hardly open a periodical without coming across the statement that what our civilization needs is more 'drive', or dynamism, or self-sacrifice, or 'creativity'. In a sort of ghastly simplicity we remove the organ and demand the function. We make men without chests and expect of them virtue and enterprise. We laugh at honour and are shocked to find traitors in our midst. We castrate and bid the geldings be fruitful.
I'm actually quite disappointed by the book which talks about parasites and predators but very conspicuously fails to take a note of the biggest parasite (and security threat) of them all. I was *very* tempted to stop reading (and I practically always finish what I started to read) after first few pages. The failure to apply one's own logic to the material being described does significantly lower my estimate of worthiness of its conclusions.
@averros - Who is the biggest parasite? :)
"How do we ensure that society's parasites don't destroy society's systems?"
Well, for starters, stop electing them.
eh foreigner, im thinkin its the government/religion industrial complex.
read buckminster fuller about outlaws, the people who discover new ways to do things because they are not experts who could explain why its impossible.
I have some trepidation about starting this book. My impression is this is very close to the root cause of most/all of humanities real problems. (Personal summary, way older than the book: "Too many assholes", not that defectors are necessarily ones, just too many are. And it takes only a small number of assholes to ruin it for everybody.)
So my fear is this: If we actually now clearly see what is wrong, but are still mostly powerless to do anything about it, would that not be close to hell on earth? (Sorry for the religious terminology, I am a firm believer atheist.) Of course not looking is cowardly, but I am tempted to do so. Well, maybe for a few weeks ;-)
Bruce, is it common to call people parasites in sociology - isnt this insulting? Also the pondering on parasitic self-interest as crossing group welfare, in the way presented, does hardly reflect, what economists have held for a long time. Think of Adam Smith. The differences are subtle, beware, and a philosopher or psychologist might help explain. Diane Coyle covers the economic aspect under the "rationality" umbrella in her talk: http://www.stifterverband.info/veranstaltungen/...
..ay to... uh... the mathematics.
..and then on to an xkcd!
"And it's defectors who are fueling the Occupy Wall Street movement. Without defectors, society stagnates."
Hmmm. . . . so although seemingly obvious, it should be overtly noted that how one views any given defector will be highly influenced by one's politics. Broad societal generalizations are difficult to make and to agree on. Sort of challenges what seems to be the premise of this book.
I might have written: "And it's defectors who are fueling the Tea Party movement. Without defectors, society stagnates."
But then most of the people on this board would probably consider me the "asshole" defector in need of censure. While I, rather, consider those of the "Occupy" ilk to be so. Who's right? Which of us is the positive "defector" that helps prevent stagnation and which is the dangerous threat?
In the first instance, I don't see Bruce's work -- although necessarily touching on the political sphere -- in political terms. He uses terms like "parasite" and "defector" to denote roles that are well defined. I understand them as distinctions whose significance doesn't depend on moral valuation.
Let's try an analogy: Without mutations, species stagnate. This is (at least within certain limits, I think) biological fact. From a scientific standpoint, its truth does not depend on any value judgment (good vs. bad) on any particular mutation, or on mutations as a category.
For me, the "Tea Party" is a splendid example. Personally, I find several attitudes and conclusions associated with that movement to be either deeply suspect, or badly flawed. At the same time, I consider the movement to be beneficial to the USA.
I see OWS in essentially the same light.
I don't have to sort everything into "the righteous" versus "the wrong." Both of those American movements, which are certainly defections from the customary political equilibrium, have increased public consciousness, and encouraged citizens to question notions they had previously accepted uncritically. I expect the USA to better for it.
Hi Bruce, without having read the book: Did you also take into account all the literature concerning the "political economy", "social capital" and the creation of public goods? (still a bit unhappy about some of those terms, but hey...)
I think almost everything ever written&researched by Mancur Olson might fit in pretty neatly with what you appear to be going at.
@Foreighner - what Bruce consistently misses (probably because he spent too much time in the artificial world of academia where the normal economic laws do notapply) is that there is very simple rule for discriminating between the predators (aka parasites, or "hawks") in the human world and the productive class (aka "doves", or cooperators). It is unilateral use of force (or fraud, or threats of force) to obtain resources from others as opposed to voluntary exchange and cooperation.
The biggest initiator of force against peaceful population is its own government - after all, the very existence of government requires taxation - i.e. taking the resources under the threat of force. It also happens to be the historical fact that all massive human-made disasters (first, wars, then genocides and ecological catastrophes) are directly initiated by the governments or enabled by them. An average human is threatened and robbed by his local overlords far more than by any other robbers and fraudsters.
As Bruce contends, more successful parasites tend to preserve life and, to a some extent, well-being of their host organism. That is generally the case with "moderate" governments. This still does not mean that the hosts won't be much better off without parasites.
Thus prominently and repeatedly listing tax avoidance as an example of security threat, defection, or non-cooperation makes one wonder if Bruce really thinks himself as being one of the ruling class (because for the rest of us tax avoidance is just an example of successful defense from the thievery writ large).
This also demonstrates the profound lack of understanding that co-operation does not necessarily involve "sharing" or altruism or "pitching in" or reciprocity; in fact the mechanism of property rights facilitates voluntary cooperation of purely self-interested individuals, and is both sufficient and necessary for a large human society to prosper.
Just like economics, security analysis cannot be wertfrei (value-free). Trying to do that basically leads to glaring intellectual dishonesty - such as failing to "notice" the largest and most vicious predators in existence.
> An average human is threatened and robbed by his local overlords
> far more than by any other robbers and fraudsters.
In your worldview, is it possible for those robbing overlords to be board members and vendors of a private corporation acting under the authority of some-document-called-a-contract that will be enforced by the courts as a contract, even if the consent of one party is a legal fiction?
@The_Right ... - a private party engaged into violation of property rights of others is criminal. If it operates under the color of "law" which allows it to force others to be a party to a "contract", this does not change that basic fact. There's such thing as unjust laws, and the US legal system was supposed to be able to take such laws down (as unconstitutional or as nullified) - but, being the part of the same gang, the DoJ and SCOTUS are quite happy to give the legislative and executive branches free rein.
I happen to think that there's only one unique basic system of law which does not create distinct classes of people with wildly unequal rights - and which does not contain self-contradictions. The corollary is that legislation is useless and stupid activity - any "laws" they pass are either tautologies (if they are good) or illogical (if not actively malicious). It follows that democracy as a concept is completely moronic. Why would anybody need voting and other voodoo of democratic rituals when one can simply apply logic to arrive to the same system of laws? (In all fairness, the proof of existence and uniqueness of the natural law was offered only in 1970s, by Rothbard, - and because it is logically sound, the reaction by the legal and sociology scholars was to pretend that it does not exist and hope that it'll go away - but a lot of other people took notice, hence the renewed interest in libertarianism today, with Rothbardian fraction taking the lead).
You think predator and parasite are synonymous? And that "the productive class" and cooperators are synonymous?
I'm not sure what you're trying to communicate but you're not doing very well.
I like the phrase "the productive class". It sounds meaningful even though, without a definition of "productive" it isn't.
Are rulers productive? Are you pushing anarchy?
Are educators productive? Are you pushing stupidity?
Are farmers productive? Are you pushing a form of communism?
But we do not agree on who legitimately deserves (owns) what property or rights. That is the essence of political dispute, yes?
You think I'm a parasite taking your stuff by force and fraud. But I think you're a parasite claiming my stuff by force and fraud. Who gets to decide what happens next?
Ooh, I missed the remark about Rothbard. Suffice it to say that the opinion that Rothbard's conception of rights assignment is acceptable and uniquely acceptable is... fringe, to say the least.
For one thing, it is wholly dependent on the physical existence of an independent and indisputable way of gauging the priority of claims and the commitment of harm, and it has no defense against monopoly enforced by contract agreement (the nullification of which is the modern capitalist state's primary defense against robber barony, not antitrust legislation itself) - incumbents at any given time always have a legally-enforceable incentive to entrench themselves and all their upstream and downstream business partners.
You can be defector/innovator in natural science, but conformist in society.
In Germany (1933-1945) were zero democracy, but German innovations (e.g. rocket science) of that period were utilized for many years after 1945 by USA & USSR.
Apart from their metaphorical use as stigmatic labels, parasitism and predation are quite distinct in biology (the domain from which these terms derive their meaning).
These terms say NOTHING WHATEVER about right/wrong good/bad should/shouldn't. They categorize types of relationship. Any biologist who talked about such relationships in terms of rights, or justice, would soon be shunned by the scientific community.
As I posted before, I don't read Bruce's ideas as being essentially political. But regular readers of the interwebs know, that almost any topic whatsoever can be painted in political colors.
There are lots of people who claim expertise in political science, law, etc. The central focus of Bruce's blog is security -- no?
And his new book is an exploration of the concept and workings of trust, in the hope of promoting better understanding. As far as I understand its purpose, it is not an attempt to describe the ideal form of government, organization of society, theory of justice, etc. etc.
PS I seem to read a critique of ideas by the ad hominem "he spent too much time in the artificial world of academia where the normal economic laws do not apply."
If Rothbard's ideas are valid -- the the extent of having been proved! -- may we then presume that they were founded on his personal experience of administering major bodies of government, managing competitive industries, and so forth?
Bruce, anyone that told you that they thought the "dishonest minority" was talking about racial minorities was someone that you shouldn't have given the book. They were just looking for things to make a fuss about. Well that or as I've often stated, the educational system has failed us and political correctness is exacerbating the issue.
averros, I must confess a great difficulty in following your train of thought. You characterize taxation as theft, but then seem to assume some powerful and neutral force that enforces property rights. How exactly do you imagine this force will sustain itself?
If a group of people get together and mutually agree that they will share the cost of some mutual protective force, and then one of them tries to secretly avoid paying the share he agreed upon, that's pretty clearly fraud.
So if you take use of fraud as a sign of a parasite, that means that tax evaders are parasites, right? The only non-fraud options are to pay your share or to openly refuse to take part in the mutual defense pact.
Isn't it Henry Waxman? I know Greenspan's an Alan, but I'm pretty sure Waxman's a Henry.
Isn't it Henry Waxman? I know Greenspan's an Alan, but I'm pretty sure Waxman's a Henry.
averros at March 4, 2012 8:30 PM wrote:
> what Bruce consistently misses
> (probably because he spent too much time in the artificial world of academia
> where the normal economic laws do not apply)
Bruce Schneier "is the founder and chief technology officer of BT Managed Security Solutions, formerly Counterpane Internet Security, Inc."
Contrast with your hero
averros at March 5, 2012 5:16 AM wrote:
> Rothbard, - and because it is logically sound,
> the reaction by the legal and sociology scholars was to pretend that
> it does not exist and hope that it'll go away -
> but a lot of other people took notice,
> hence the renewed interest in libertarianism today,
> with Rothbardian fraction taking the lead).
"From 1963 to 1985, [Murray Rothbard] taught at Polytechnic Institute of Brooklyn, later to become part of New York University in Brooklyn, New York. From 1986 until his death he was a distinguished professor at the University of Nevada, Las Vegas."
Are you going to tell us that Murray Rothbard -- who lived in the artificial world of academia for 30 years -- has more real-world experience with "normal economic laws" than somebody who actually started and still runs a business?
The cost (in all terms) of security should be mentioned here in the context of return on investment. My favorite example is the premise that corruption should be rooted out until the remainder is an incidental expense that would cost more to completely remove than the resulting benefit would justify. The first people to experience transparency should be government policy makers and executives. Give all them brain scans, starting with the most powerful positions first, to make sure that they are not sociopaths or narcissists. The people with malfunctioning internal moral compasses and those with arbitrary powers are the most likely and most costly defectors.
@ those that consider governments to be the worst predator of all: read the comments above yours. *You* are the political outliers, with a closed mind of your own choosing. Indeed, you strive for an anarcho-syndicalist society which is on the far left of communism, because that is what slim-to-none government begets. Governments aren't there for nothing, just make sure you choose the right ones. Otherwise, you'll end up complaining about them. So... you didn't choose the right ones ..?
@Bruce - Your essay has provided me with ample reason to purchase your book... (done)
To float a suggestion for your next book (or at least some spirited discussion):
Until food and energy scarcity are addressed, there are strong incentives (chiefly, survival) for individual and collective parasitism.
Assuming that problems of food and energy scarcity can be solved, what path of technological and cultural development will optimize for group survival despite parasitism?
Does the present course of technological and cultural development parallel this ideal course?
If not, can the present course of technological and cultural development reasonably be ascribed to Hanlon's razor - or are parasitic elements of society prosecuting a conspiracy against humanity at large?
Schneier.com is a personal website. Opinions expressed are not necessarily those of BT.