Schneier on Security
A blog covering security and security technology.
« More RIPA Creep |
| Comparing Cybersecurity to Early 1800s Security on the High Seas »
April 16, 2008
Our Inherent Capability for Evil
This is interesting:
What took place on a peaceful Californian university campus nearly four decades ago still has the power to disturb. Eager to explore the way that "situation" can impact on behaviour, the young psychologist enrolled students to spend two weeks in a simulated jail environment, where they would randomly be assigned roles as either prisoners or guards.
Zimbardo's volunteers were bright, liberal young men of good character, brimming with opposition to the Vietnam war and authority in general. All expressed a preference to be prisoners, a role they could relate to better. Yet within days the strong, rebellious "prisoners" had become depressed and hopeless. Two broke down emotionally, crushed by the behaviour of the "guards", who had embraced their authoritarian roles in full, some becoming ever-more sadistic, others passively accepting the abuses taking place in front of them.
Transcripts of the experiment, published in Zimbardo's book The Lucifer Effect: Understanding How Good People Turn Evil, record in terrifying detail the way reality slipped away from the participants. On the first day Sunday it is all self-conscious play-acting between college buddies. On Monday the prisoners start a rebellion, and the guards clamp down, using solitary confinement, sleep deprivation and intimidation. One refers to "these dangerous prisoners". They have to be prevented from using physical force.
Control techniques become more creative and sadistic. The prisoners are forced to repeat their numbers over and over at roll call, and to sing them. They are woken repeatedly in the night. Their blankets are rolled in dirt and they are ordered painstakingly to pick them clean of burrs. They are harangued and pitted against one another, forced to humiliate each other, pulled in and out of solitary confinement.
On day four, a priest visits. Prisoner 819 is in tears, his hands shaking. Rather than question the experiment, the priest tells him, "You're going to have to get less emotional." Later, a guard leads the inmates in chanting "Prisoner 819 did a bad thing!" and blaming him for their poor conditions.
Zimbardo finds 819 covering his ears, "a quivering mess, hysterical", and says it is time to go home. But 819 refuses to leave until he has proved to his fellow prisoners that he isn't "bad". "Listen carefully to me, you're not 819," says Zimbardo. "You are Stewart and my name is Dr Zimbardo. I am a psychologist not a prison superintendent, and this is not a real prison."819 stops sobbing "and looks like a small child awakening from a nightmare", according to Zimbardo. But it doesn't seem to occur to him that things are going too far.
Guard Hellmann, leader of the night shift, plumbs new depths. He wakes up the prisoners to shout abuse in their faces. He forces them to play leapfrog dressed only in smocks, their genitals exposed. A new prisoner, 416, replaces 819, and brings fresh perspective. "I was terrified by each new shift of guards," he says. "I knew by the first evening that I had done something foolish to volunteer for this study."
The study is scheduled to run for two weeks. On the evening of Thursday, the fifth day, Zimbardo's girlfriend, Christina Maslach, also a psychologist, comes to meet him for dinner. She is confronted by a line of prisoners en route to the lavatory, bags over their heads, chained together by the ankles. "What you're doing to these boys is a terrible thing," she tells Zimbardo. "Don't you understand this is a crucible of human behaviour?" he asks. "We are seeing things no one has witnessed before in such a situation." She tells him this has made her question their relationship, and the person he is.
Downstairs, Guard Hellmann is yelling at the prisoners. "See that hole in the ground? Now do 25 push-ups, fucking that hole. You hear me?" Three prisoners are forced to be "female camels", bent over, their naked bottoms exposed. Others are told to "hump" them and they simulate sodomy. Zimbardo ends the experiment the following morning.
To read the transcripts or watch the footage is to follow a rapid and dramatic collapse of human decency, resilience and perspective. And so it should be, says Zimbardo. "Evil is a slippery slope," he says. "Each day is a platform for the abuses of the next day. Each day is only slightly worse than the previous day. Once you don't object to those first steps it is easy to say, 'Well, it's only a little worse then yesterday.' And you become morally acclimatised to this kind of evil."
EDITED TO ADD (5/13): The website is worth visiting, especially the section on resisting influence.
Posted on April 16, 2008 at 6:40 AM
• 70 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
It's even more twisted when the White House claimed to be seeking 'moral clarity' on the definition of 'cruel, inhuman, and degrading treatment.'
It's not so much a line as a one-way turnstile. You can't easily cross back without going out of your way.
Experiments like these, in particular Milgram's series of experiments are the reason why we now have Human Subjects in Research boards.
Human Subjects restrictions are becoming a major problem in studying security usability as the only way to reliably gather data we know of at the moment is to attack the subjects without their knowledge. As a result some of the work can only be done out in industry (e.g. Mary Ellen Zurko's studies on people working for a company that develops software for an email program).
But the Zimbardo experiment was probably the one that caused the government to act. Inconvenient results.
I always use this as an explanation why I dont like the way the current long-term trend in the US, where anything that is not prohibited is compulsory, is causing us to become buried in police agencies of one form or another (State, Federal, Local, County. FBI, TSA, IRS, pretty much any other combination of TLA).
If you have a huge sub-society that has the power to harrass the rest at will, they unintentionally become sadists. (Granted, not all of them; just the 60% or so who are normal. The bottom 20% probably already were sadists, or at least criminals who joined the massive panic infusion of police hiring in order to facilitate existing criminal activities. So only the 20% saints at the top may resist this mentality.)
Its absolute proof of the "power corrupts" cliche.
Another good one is the Milgram experiment, it shows a Holocaust event ("I was just following orders") could easily happen here:
Makes for an interesting perspective on the "Security guy being a dick" stories, moving the focus back on to "why does he think he's being resonable? Why does his organisational culture lead him to behave this way?"
Thinking about the Jefferson Memorial incident, some have already commented, of all the multivarious police forces in DC, the Park Police have the worst reputation for unthinking, heavy handed intervention. Maybe the debate should step back from "Did the cops do the right thing?" (and you have to really distort the scenario to give an unequivocal yes to that) to "Who is in charge of the Park Police and why do the PP think this is what their job means?"
Understanding How Good People Turn Evil ? Why does everyone insist that good people might do things like that eventually ?
In my eyes, they aren't good - good people behave even when there aren't any external sanctions or rules. Zimbardo's prison guards or Lynndie England are at best amoralic, at worst evil.
Oh please, it's no WONDER his experiment turned out evil. I mean, LOOK at the guy! He's clearly an Evil Genius. He could open a bakery and it would turn out evil...
They need to do this experiment a few more times, each run by people who DON'T look like Ming the Merciless...
Remember, these were your everyday bright young students. If you had told any of them a few days earlier what they would end up becoming, they would have denied it fervently. Chances are high that any given person would be either one of the silent or one of the sadistic. Even you.
You're reacting, not thinking. The whole point of the experiment is to demonstrate how large populations of otherwise normal people can be complicit or active in things like the Holocaust. How societies can be evil even despite the individuals that compose them. It's absolutely essential that we understand this, so that we can better avoid such atrocities.
Think about it a little longer before you dismissively label the majority of the human race "evil".
Somewhat related, another great example of what happens when a small group of humans is isolated from the rest of humanity is described in "Losing the Plot", by Mark Jeffery.
He relates the (supposedly true) story of how the leadership of a British antarctic base, cut off from home by winter, starts making decisions that appear completely irrational, endangering the lives of Jeffery and his colleague by insisting on "sticking to the rules".
I'd be interested in finding out more about the volunteers. The description "bright, liberal young men of good character, brimming with opposition to the Vietnam war and authority in general" isn't enough for me to judge the experiment except to say the psychologist is pretty f'ed up for doing it in the first place.
It's pretty easy to sit back in relative comfort and say that a person who claims to be "good" but does something "bad" isn't a Good person. And maybe in this case, it's the truth (again, we don't know enough about the volunteers.) However, in general, "good" people do "bad" things. Even if it's just yelling at the airline ticket counter person for your canceled/delayed flight, or the barista for screwing up your coffee order.
To be good, as defined by erlehmann "good people behave even when there aren't any external sanctions or rules" is pretty much impossible to achieve especially in such an extreme situation. We're all "bad" at one time or another. My personal opinion is that a "good" person regrets doing their "bad" things and tries not to repeat them.
"Why does everyone insist that good people might do things like that eventually ?"
Because we tested it, and they did. That's what the experiment was for! The experimenters picked people at random, put them in a specific situation and environment, and they all behaved "evil." Remember, these were idealistic students who opposed war on moral grounds. The Milgram experiment (linked above) was repeated many more times with a wider variety of people.
Before hearing any evidence, most people would agree with you, which is why we have science instead of just believing our first instincts. It turns out your first instinct about good people is wrong in one of two ways:
1. Most "good" people do "evil" things in the right environment
2. Most people are "evil."
Take your pick. Most people think #1 is most plausible. Picking neither means you resign yourself to irrational cognitive dissonance or have a poor understanding of probability, but I guess that's technically a third choice.
The experiment does _not_ suggest that people always want to do these things, deep inside. It only suggests that roles and environments such as "prisoner/guard" or "just following orders" can _cause_ that behavior in most (otherwise good) people.
"Are we, ultimately, incapable of Humane government? Is there something about us that lead us away from Humanity.
Chuang Tzu denies a common human nature, but he does say this:
Games of skill and cleverness begin in a light mood, but they always end up dark and serious. And if things go far enough, it's nothing but guile. Drinking at ceremonies begins orderly enough, but it always ends up wild and chaotic. And if things go far enough, it's nothing but debauchery. All our human affairs seem to work like this. However sincerely they begin, they end in vile deceit. And however simply they begin, the grow enormously complex before they're over. (55)
From the Blog "Useless Tree"
You have said it yourself: "Chances are high that any given person would be either one of the silent or one of the sadistic. Even you."
I agree - exactly that makes me say that people aren't "good" - most persons I know cannot even clearly define their moral standing and many of them who say they know what is good aren't even trying to act according to that. Lots of people can be persuaded do to things by use of good rhethoric. Many cheat and lie when getting the chance.
Only a minority holds their beliefs till someone counters them in a discussion, most just do what pleases them and otherwise follow the herd. A friend of mine once joked she probably would have been a good nazi, as she observed herself behaving like this.
It's a little bit like eating junkfood - the majority doesn't care about it being unhealthy, even if they know about it. DIfference here is that the damage is done to one's self, so there is an essential motivation to stop - and they still do it.
On the face of it, I cannot explain why there still are "good" people.
The mentalist and performer Derren Brown does an interesting Milgram experiment, and also does an experiment where he programs people to commit a bank robbery called ‘the heist’ based off of visual and audio triggers. Its an interesting watch, youtube it.
Ive always thought that if you put people in the right situation you can turn them into anything. Good or bad.
Ah, the classic no true Scotsman ripostes to the classic psychological studies.
Most people succumb to normative pressures. That doesn't make them good, evil, or anything else. It just makes them human.
We can all pretend we'd act other than how those students did, or even how Lynndie England did, but until we're put in that situation, we can't know.
Read "Cracking the Code" by Thom Hartmann. There's much to digest in there on the subject of the inherent evil in man.
Remember that this was not a prison, it was a college "game" with people playing the role of what they thought guards were. People already believing that guards, and any authority figures, were supposed to be evil.
Interesting for sure, but a measure of what liberal students would do, not what normal people would do.
We control our police and prison guards bu setting limits on what they can do. Many of the events described in the test are simply not allowed. The honest ones can call on authority to control the rest.
Excellent german movie on this topic: "Das Experiment", just about the described plot. Hard to get in english though ...
A couple of thoughts. Under the right stressors, many folks will do bad things. Yet, other folks will not, no matter how bad it gets.
We are programmed to be pack animals and follow the leader. Most folks have a tipping point where they will break away from an increasingly evil leader. Unfortunately, that tipping point is moderated by personality, maturity, and rhetoric. Resisting the Jedi (Sith) mind-trick, if you will.
Most people, when put in charge of other people, prove to be bad choices. The key that unlocks the bad potential is having power over others. When everybody is powerless, we're all in the soup together, our brother's keepers. When somebody gets power -- enforced by violence -- we're no longer all together anymore.
Why else would the vast majority of supervisors turn out to be insufferable heartless sadistic pricks?
This handily explains the existence of "griefers" as well. The normative structure in virtual worlds is either premised upon giving a shit about your standing in the virtual world, or is exceedingly weak for other reasons. Result: people get off on making "life" miserable for others, and do not have to pay for it in any currency they care about.
In the prison experiment, the normative structure was either inverted or uncertain.
We the germans, the turkish, the hutu (and many others) are waiting on the road to hell we walked so foolishly, to guide the proud US on their way....
"Evil" is an imprecise and emotionally loaded term, that brings to the discussion the usual hazards normally associated with Godwin's law.
What this experiment seems to bring into focus is the fact that the capacity for tyranny, whether grand or petty, can be found in almost anyone. Control and coercion appear to be near-instinctive in many --- possibly most --- humans. On reflection, this can hardly be a surprise given the historical frequency with which tyrannical governance emerges even from idealistic and religious movements.
It is worth remembering, however, that one may also observe the obverse side of human nature: there exist conspicuous examples of moral behavior under extreme duress. Andrei Sakharov is one example --- someone who while living in a system notorious for liquidating its civil opposition, deliberately chose public scorn, imprisonment, and exile over honors and material rewards, entirely out of personal moral conviction.
There are many other such cases of individual moral heroism, I'm sure everyone has a few favorites of their own. They at least give one something to aspire to and hope to emulate, even for those of us who might secretly doubt that we have the resources of character to resist becoming one of Zimbardo's guards (or worse) should suitable circumstances arise.
Have these experiments ever been repeated (particularly by someone who is skeptically inclined towards the original findings?) One or two experiments does not a finding make, nor would I want to base my view of human nature or social policy on such a fragile limb.
It seems we experience less extreme examples daily: YouTube videos of teens beating each other and less obvious "bullying" of coworkers in the peer group environment.
In all this, I think the best to come of the research is, "Zimbardo is now researching heroes such as Darby, 'ordinary people who do extraordinary things when other people are doing bad or doing nothing.'"
I believe it is important to understand and accept what we are and how we act but more important to understand how we can break the cycle.
If Zimbardo can help us to understand the catalyst for change perhaps we can learn to stop atrocities quickly.
You might as well ask "why are people greedy" at the same time as asking "why are people evil" the reasons appear to be the same.
Human society grew out of several thousand years of being "hunter gatherers" and most people just accept that as a statment with out thinking about it...
Hunting is inherently social as it usualy takes a group of people with a common interest (hunger) to defeat a stronger opponent (ie the prey such as an antalope etc).
It is in the interest of a hunter with special knowledge such as the likley location of prey to enlist the help of others to exploit a resource that they would otherwise be unlikley to be able to exploit as an individual.
However once the prey is downed a different perspective may well take over. This can result in a free for all as the social participants revert to gatherer type "me first type activity".
Gatherer behaviour is based around guile and secrecy. Basicaly if you know of a resource such as a bush with ripening beries it is in your own best interests not to allow others to know this.
In the gatherer case knowledge is power and you can chose who to reveal or not reveal it to.
As a gatherer you as an individual can fully take advantage of the reasource there is no requirment for social interaction other than to trade for other reasources such as mating priveleges or protection etc. The only requirment to maintain your status is to use guile to protect your resource as a secret.
The hunter on the other hand must enlist the help of others to exploit a resource. However they do not need to reveal how they know where a resource is likly to be.
Therefor an individual who can repeatedly bring a group of people together to exploit a resource is more likely to enjoy privaledges that others who can not so do. We inveriably give this sort of person status as a leader based on their percieved wisdom or luck, often casting them out painfully when they fail (build them up knock them down behaviour of the masses).
Now for the bit where I get shot at 8)
Most sucessfull leaders apear to have psycopathic tendencies (ie lacking in morals) and this appears to be a success trait as it alows them to focus on achieving a goal with little distraction.
However it does not make them sociopaths as they usually need the participation of others to achive the goal.
Those that have both traits are often refered to as "lone wolves" and they are limited in the goals they can achive.
Those that most would call psycopaths are in fact lone wolves who have no moral objection with turnning on their own.
Most people do not have sufficient of either tendency to be directly successful, however they still crave either recognition or status, for the privaledges and it gives them.
The priveledges are things that people who taste them are loath to give up, and will try to hang onto with all their abilities.
Those often described as liberals are actually seaking group approval to achieve status and recognition in the group by pandering to the percieved group wisdom (kind of I'm holier than thou attitude). However these people realy possess the skills to be leaders (thats why they are sometimes known as the "chattering classes").
However put a person without the natural skills randomly in a position of arbitary status over others and they will be loathed to give it up (power addiction).
The experiment actualy encoraged the "guards" group to come up with a new group view which was not in any way liberal. However it provided recognition and status which the individuals craved so satisfied their need.
However they still had the "I'm better than you" ethic as their success model only this time it was "evil" not "holy".
So the group dynamics had not changed nor had the real ethics just the chosen group goal.
Those we refer to as good often have strong morals, but may or may not possess the social skills to be successfull. Often though success is actually unimportant to them as their moral outlook tends not to alow them to be addicted to power.
I wonder if the experiment would have different results with a more diverse (in particular with regards to age) group. When I was in college, it was clear that a larger percentage of people were not ready for marriage, primarily because they were not independent. Does this skew the individual's ability to oppose abusive treatment? It can be very difficult to compose a diverse experiment, so I understand why it would not be done, but it suspect it changes the results.
As someone who has played "role-playing" games before, taking on an "evil" role, I've observed my attitude and thoughts (in real life) started shifting subtly according to the personality I expected myself to play in the game. So I'm not shocked by the findings.
But I would be curious what orders or expectations were placed upon the guards at the outset of the experiment. This seems like a highly relevant "initial condition" that would shape the outcome of such an experiment.
These people may have behaved in a "human" way. But that does not excuses the behavior any more than its excuses the governments.
Surly we don't need to debate the morals of forcing people to "pretend" bum F**K each other.
After catholic high school and the armys 91C school at letterman hospital in 68, I was assigned to the 1/20bn eleventh light infantry, amercal in 1968 about 6 months after they had committed the My Lai massacre and some others.
I was a neophyte with all the altrustic training suddenly in a place where morality as I had been taught was completely turned upside down. The company was kept far from the lowlands, back in the jungle to keep them away from reporters. I did not know the history of what went on before I arrived, they never tell the new guy about that stuff, but you overhear some talk and you know its bad. after a period of time in a world where people get shot occasionally, where cruelty is a common behavior, people turn animalistic, it happens in the military very easily, the training is entirely conducive to it.
At one point, I was about to photograph some prisoners who were being tortured with a TA12 field telephone, someone I had never seen before came over and told me that if I took the photo, I would be killed, and he wore the same uniform that I did. I assumed he was military inteligence, a strange animal to the normal trooper. I did not take the photo, I had not thought of this problem before, after that, i realized that whenever some criminal authority figure tells me not to take a photo, that is exactly the photo to take. No one is prepared by our society to resist authority, you have to learn that for yourself.
My time in the 1/20 was enough to realise that anyone can be reduced to an animalistic morality very quickly. To resist authority is something you have to prepare for by experience. For the 6 year that the moronic media sucked up to georgy boy the way they had sucked up to kenstarr and newtgingrich before.
I think they are now afraid of Obama, as he strikes the right note of maturity and thoughtfullness that will challenge all the false bushit that we have been force fed by the corrupt media. his statement using the word bitter was an acceptable statement, Chris matthews, the clinton backer with the squeaky voice that is always interrupting better people chewed on the word bitter for five minutes and called it patronizing and condescending, then he patronizingly said that the voters in pennsylvania had made up thier minds on how they are going to vote in this election in 1957. How patronizing is that? We are very ill served by these corporate shills who cannot be voted out of the media.
In some television show about this incident, I recall that Guard Hellmann was intentionally pushing the envelope to see how much he could get away with.
He knew what the experiment was about and decided to take it to whatever extremes the doc would allow since they were really given no boundaries on what their roles were (beside the generic "prisoners" and "guards").
In the absence of a force like Guard Hellmann, things probably would have been bad, but not bad enough to end the study early.
It's interesting that the talk of evil hasn't given rise to much talk of good. If you know anything about the religions of the world, several tend to hold saints in high regard for doing extraordinary good in the face of evil - so much so that the religions believe these people have been given a pass in the afterlife. These saints are held up as life examples for followers of their particular religions to emulate in any given situation. Mother Teresa is a somewhat recent example of someone on the path to sainthood: http://en.wikipedia.org/wiki/Mother_Teresa
While religious leaders often fail by acting counter to their religions' central tenets, some religions do provide a strong moral code that, if truly followed, would prevent the actions witnessed in the experiment. It would be interesting to see this same experiment conducted at a religious school.
Even if you don't read the book, the Lucifer Effect Web site is worth a visit:
Especially the "Resisting Influence" section.
"...some religions do provide a strong moral code that, if truly followed, would prevent the actions witnessed in the experiment."
Unless, of course, the name of the game is changed, from "Guards and Prisoners" to "Faith and Heresy". You can find extensive descriptions of that experiment in histories of the Counter-Reformation. That's just one of the most spectacular examples, of course --- there are many others, some still in progress today.
The point is, this isn't about "evil". It's about tyranny, and coercion. Which arises about as often for "good" ends as for "evil".
The true sadist was the person running the experiment. He should have stopped it after day one, or at least made the prisoners and guards change positions.
Ever wonder why we keep seeing these prison and prisoner shows on the NG channel? Are we being alerted to what is coming or de-sensitized what there will be more of?
While it is true a minority of people can and do perform evil deeds, the larger question is, why do the majority find it so difficult to intervene and stop them? The answer can be found in this fascinating book:
Influence: The Psychology of Persuasion
by Robert B. Cialdini
I remember running into discussions of this kind of experiment in college psych classes. The discussion there centered around one intended to find out just how far an ordinary person could be pushed before they'd refuse. The story given to the subjects was that they were helping in an experiment on how punishment influenced truthfulness. They were to ask questions of another test subject, and give them an electric shock if they didn't give the correct answer.
What they weren't told was that the "subject" they'd be shocking wasn't hooked up at all, the only thing wired up was a meter showing the shock level and the "subject" would fake an appropriate reaction per a script they had. The researchers would push for higher and higher shock levels, seeing how far someone would go before telling an authority figure they wouldn't do that to another person.
The reported results: most subjects were reluctant and uncomfortable, but most would also continue to increase the shock level past the point clearly marked "Lethal Voltage" on the setting dial with relatively little urging, even when the "victim" being shocked had collapsed and wasn't responding to questions or shocks anymore.
Sadly, even back then those results didn't suprise me. Most evil isn't done by evil people, but by ordinary otherwise-decent people who aren't capable of telling an apparent authority figure "Sod off. That's wrong and I'm not doing it.".
In 4th grade, my school conducted a similar 'experiment' for a day -- all classrooms participated. Half of the kids were made to wear blue collars of construction paper and were treated as if they were being punished while the other half were given undeserved praise and rewards (cookies, playtime) and were encouraged to look down upon those wearing the collars.
Then, after lunch time, the teachers made everybody switch. Now the blue-collar wearers were the privileged and the collarless were punished.
For the last hour or so of the day, the collars came off and we were all equals again and there was a lot of discussion.
The end result? At the time, it was just one more weirdo thing teachers made us kids do.
In retrospect, especially in the context of the milgram and zimbardo experiments, its obvious that the teachers were trying to give us the perspective of how easy it is to assume the role of oppressor with all the privileges it comes with.
The US does not condone this sort of behavior in psychology classes.
However, I plan to issue a signing statement that refutes my need to abide by the above statement.
There are a number of related studies, involving getting people to ignore obvious danger or mis-see obvious things in front of them based on social pressure. The conclusion I take from those studies is that we are surprisingly susceptible to having our perceptions of right-and-wrong, prudent behavior, and even reality changed by the people around us.
The useful thing to take from that is that you are subject to this, in the same way that you're subject to making bad decisions under the influence of alcohol or sleep deprivation or anger or horniness. This makes it a lot more likely that you can avoid going along with some widespread mistreatment of someone or evil that seems reasonable because everyone else is doing it. Presumably the people who *didn't* go along with evil in the past were not immune to social pressure, but instead were aware that social pressure could lead you to do bad stuff, and so they were able to resist that tendency.
@Jason: "In the absence of a force like Guard Hellmann, things probably would have been bad, but not bad enough to end the study early."
That's interesting, but does it really change the results of the study? It doesn't look like any of the other guards tried to stop him. If anything, I'd say he helped the study along by showing off the more extreme elements of his fellow guards' nature in a way the experimenters alone could not have.
Also, it seems like he's a bit of a douche.
This problem is a constant challenge for those who provide leadership for the protective service professions: police, military, security, corrections, and so on.
One problem is to keep the chain of command in charge and reject outside influences, typically corrupt in nature.
Another problem is to keep challenging our assumptions or to make sure that someone challenges them for us (usually but not always the courts.) This is why habeas corpus and the rights of prisoners, civil / military / otherwise, is so important.
A third problem is to be sure that leaders and especially command officers know, as a matter of fact and not of theory, that ANYONE is capable of the behaviors shown in the Stanford Prison Experiment, at Gitmo, in Iraq and in Vietnam, and in death camps from Poland to Siberia to Cambodia. It is the task of leaders to keep their subordinates within the bounds of human decency even under extraordinary stresses.
Last but not least, the lay public does not always appreciate the idea that the bulk of the force used by police, military and especially corrections is ugly and brutal but lawful and necessary.
Certainly minimum force should be employed, but ultimately, the alternative to guards using force is to allow the inmates to take over the prison. Or criminals to take over the society.
The balance point is between training people to be swift, effective and safe in using force as part of a unit with esprit de corps and cohesion; yet composed of individuals still willing and able to challenge, to constantly question the line between war and atrocity, between policing and politics, between corrections and corruption.
Not an easy task, but essential to the preservation of human rights and liberty.
Yes, exactly. The thing that plays the biggest part in making you susceptible is believing that your superior education, intelligence, morals, genetics, or whatever will render you immune to social manipulation.
(Of course this warning goes double for everyone who's read this far, because by now you probably think you've learned something about it!)
"But, wondered Thomas Carnahan and Sam McFarland, what sort of students volunteer for such a study?
Carnahan and McFarland couldn’t test this theory using the SPE data, and instead re-ran the volunteering part of the study to see what sort of people would volunteer for a two week study ‘of prison life’, compared to those volunteering for a study described in an identical manner but with the ‘prison life’ bit taken out. This extract from their abstract tells what happened:
'Volunteers for the prison study scored significantly higher on measures of the abuse-related dispositions of aggressiveness, authoritarianism, Machiavellianism, narcissism, and social dominance and lower on empathy and altruism, two qualities inversely related to aggressive abuse. Although implications for the SPE remain a matter of conjecture, an interpretation in terms of person-situation interactionism rather than a strict situationist account is indicated by these findings. Implications for interpreting the abusiveness of American military guards at Abu Ghraib Prison also are discussed.'"
@erlehmann: "On the face of it, I cannot explain why there still are 'good' people."
There probably aren't, at least by a definition as strict as you seem to be using. It only seems like there are, because most people are never put into situations where they're pushed over their limit.
I guarantee you that if you look hard enough, there's a way to coerce anyone into becoming a monster. The right lies, the right motivation, the invocation of the right higher powers; everyone has the capability. It's just lucky for us as a civilization that most people never find out what it would take to turn them into the metaphorical Auschwitz guard. But anyone who thinks they couldn't become monstrous if put in the right situation is deluding themselves or operating under a messiah complex. You're not a special snowflake, and neither were the guards.
@Jason: "In the absence of a force like Guard Hellmann, things probably would have been bad, but not bad enough to end the study early."
There's *always* a Guard Hellmann. How many people were in this study? It doesn't sound like more than a few dozen at most. And they had him. Of course we'd need to run more experiments to be sure, but I doubt there's any reason to suspect that the study group was in any way atypical. In any group of 25, randomly-selected people, I'm sure you'd find at least one or two Guard Hellmanns.
At the very least, even without getting into a discussion of exactly how many Hellmanns there are in society, or how frequently they turn up and in what roles, the fact that people like that exist at all mean that you cannot predicate society on the nonexistence of such people. You need to build social structures with the *assumption* that there are sociopaths who will exploit them in order to push the envelope and torture others for the sheer joy of doing so. To do anything else is naive.
This is really where I think the 'security' lesson comes in. In the same way that we think of old Internet protocols as naive and silly because they didn't assume malice on the part of users, it would be foolish to build social structures and not assume malice on the part of occasional members of law enforcement, the legislature, the judiciary, or any other priviledged group.
"Nothing but good can result from an exchange of information and opinions between those whose circumstances and morals admit no doubt of the integrity of their views." --Thomas Jefferson to Elbridge Gerry, 1797
The important things never change.
I wish to avoid all collisions of opinion with all mankind.
That's interesting. I suspect that the basic result is still meaningful, though, because most people who work as policemen, prison guards, and (today) soldiers chose those lines of work. People who are willing to take a job as a prison guard are probably pretty different from most people.
This is why there has to be strong, meaningful punishment for guards who behave like that. I have to think that the 'guards' in that experiment had no prior training in how to be guards, or what sort of behavior would be tolerated. I suspect they would behave differently if they were going to be prosecuted or face any sort of disciplinary action based on their behavior. What this shows is that when someone is allowed pure power, it gets bad really fast.
It's why we have a system for justice, and no one is supposed to have unfettered power.
Mother Teresa is the same kind of sadist Guard Hellmann was. Making people go without opiate painkillers in her hospital was part of her plan to bring the suffering poor closer to god through some masochistic mechanism only she understood..
Don't hold her up as an example of a good person, she wasn't. If you had chronic pain at one of her clinics, she'd of been as bad as that guard, worse even. A Pious Sadist, with many psychosexual issues, keeping the poor in pain for Jesus. Amen.
Gotta do a lot better than mother teresa for a good person.
One problem with the experiment is that the officers did not have to obey any laws or rules themselves. In the real (developed) world you have to obey a ruleset that would preclude this type of behaviour. The reality is most prison guards don't want to give prisoners a hard time. They just want a quiet day. A lot of the prisoners want the same. Generally you don't have to like each other but you have to live together.
Of course a ruleset has to come from the top. Poor leadership or evil rules from the top and you get the Gulags, Abu Ghraib and Guantanamo and the associated abuse that comes from too much power.
Much as we may hate it, the experiment reflects a a very real phenomenon. There is a truly scary little book out there called "Ordinary Men: Reserve Police Battalion 101 And The Final Solution In Poland" about one of the special death squads used to exterminate Jews on the Eastern Front.
What is scary is not what these guys did -- which bad enough -- it's what they were. There were not Nazi fanatics or hard-core thugs. They were for the most part regular working-class Germans who systematically committed mass murder as everyday business.
I don't know about the innate perfectability of man, but we have far too much proof of how easy it is to lead people into doing utterly horrible, evil things.
It's worth noting a couple of things about the Stanford Prison Experiment:
1. It cleared all contemporary human research bodies. They looked the experiment over, and passed it without a second thought. That wouldn't happen today, of course. Today's human research committees know about the SPE. But that's the only reason! It's not because they were dangerous cowboys then, or we're so careful now... They were trying to be very careful then. The results of the experiment were utterly shocking to all concerned.
2. Virtually all of the subjects asked to be prisoners. Guard Hellmann was no exception. Most of them were typical 60s Stanford students, anti-war and anti-authoritarian, again including Hellmann. The only exception I recall (I read the book several weeks ago, but I'm not referencing it now) was a fellow nicknamed "Sarge" for his conservative bearing. I'm not sure of his political views at that time, but he clearly did not fit in with the Stanford crowd. He was also one of the only participants in the experiment to stand up for humanity and individuality.
I mention this to suggest that resistance to evil is a trait independent of political outlook.
PS @PHB: The Tuskegee syphilis study was the one that forced the government to act. The SPE was after that and had been cleared by the relevant boards.
That's not true at all. Have you read the book, or researched the experiment? The guards did have rules, most notably that they were not to physically attack or scuffle with the prisoners. One of the eerie and chilling things about the experiment is that the guards largely obeyed that rule. There were a few cases of guards restraining prisoners or shoving them into cells, and there was the famous use of the fire extinguisher, but aside from that (and in neither case did the Warden, who was responsible for enforcing those rules, comment on those actions) everything that happened in the experiment was accomplished by threat and command, not by force.
@Alex and John
Research the study. Zimbardo and his assistants didn't pull volunteers off the streets, they screened them very carefully. The point of the experiment was to find out what a prison would be like if it was full of well-adjusted people, rather than full of the usual sort of folks you'd find in a prison.
Carnahan and McFarland are either being misquoted or they're misleading us. To establish that any of their results are applicable to the SPE they'd need to show that volunteers responding to ads like the one that ran for the SPE (available either in the book or presumably from the archives of various Stanford libraries, since I think it ran in a newspaper) were unusual in a way that would have slipped past the screening done for the SPE, not just that the body from which the subjects were selected was unusual before that screening was done.
I'd like to point out several VERY serious criticisms of the Stanford Prison Experiment that seem to be surprisingly little known.
1. The results of this "scientific experiment" were never published in a peer reviewed journal, only a semi-formal abstract in a non-specialised, non-peer reviewed publication (Naval Research Reviews) several years after the experiment (and numerous subsequent for-profit books and layman's articles). For an experiment that has had such an impact on ideas about human nature, this is extraordinary.
2. Even though most of Zimbardo's raw data has never been published, his conclusions have been widely criticised as being inconsistent with the data that has been released. In particular, he describes certain behaviours among the "guards" in quite deterministic terms, as if they are an inevitable consequence of the environment, even if only two "guards" (or sometimes, only ONE) performed the behaviour and the rest opposed it.
3. At least one participant in the experiment has claimed that Zimbardo's reporting contains serious inaccuracies and exaggerations. In particular, he claims that several incidents that Zimbardo reports as being spontaneous actions by experiment subjects, were in fact scripted by Zimbardo himself with the co-operation of those subjects.
4. Due to these criticisms, there has been a modern attempt to replicate the experiment -- a touchstone of the scientific process. This experiment (which unlike Zimbardo's, WAS fully documented and peer reviewed) reached completely different conclusions. Not opposite conclusions; totally, orthogonally different conclusions.
As a result, the "SPE" (as it is widely called by those who use it) is now largely a tool of political activism; among serious psychological researchers it is regarded as horse-puckey (evidence: check for citations of the Naval Research Reviews article in psychology journals. Good luck finding any.)
"4. Due to these criticisms, there has been a modern attempt to replicate the experiment -- a touchstone of the scientific process. This experiment (which unlike Zimbardo's, WAS fully documented and peer reviewed) reached completely different conclusions. Not opposite conclusions; totally, orthogonally different conclusions."
That is a complete misrepresentation of Abu Gharib.
My take on this is that people are not naturally good or bad. Nor are there any such things as "good" and "evil", except as abstractions. Too many of us can't handle abstractions, so when they see someone do something rotten they start vapouring on about "evil manifesting itself". In fact, people are very, very adaptable. They experiment with different behaviours, and adopt those that seem to pay off. Most of us live in a stable, controlled, peaceful environment where what pays off is being polite, law-abiding, and trusting. Put us into an anarchic environment, we would be sitting ducks. Those who say "truly good people would never do these things" are expressing the belief that some of us have had "nice" behaviour so deeply imprinted on us that no change in environment could shift it. There may be a handful who are as saintly as that, but for my money the huge majority of us would adapt quickly. Spoiled children demonstrate these principles quite clearly. It's not their fault that they behave badly, are arrogant, ungrateful, and have too high expectations. That's what they have learned from their lives so far. Most people would abuse power if they had it. As Lord Acton said, "All power tends to corrupt, and absolute power corrupts absolutely".
think about it, what is the most creative and diabolical evil you could do to a person? My guess is that eveyone could come up with some horrible idea, most already have when tormented by someone. thought about what horror they would like to perpetrate on that one.
Now ask yourself, what is the greatest good you can do for another person,
My belief is that if you examine the two possiblities, good is difficult to create novel ideas and really dosent have much effect except temporarily.
On the other hand, evil is a vast area with depths beyond depths in human history.
"what is the most creative and diabolical evil you could do to a person?"
Intrude into their lives, trying to "create good".
"Now ask yourself, what is the greatest good you can do for another person,"
Leave them the hell alone?
The persecutorial response to "Heresy" is a man made concept usually, and isn't found in all religions. However, though it might not be a precept of the religion itself, it has been picked up by some followers in an attempt to create a "convert or die" situation. The sadists do it for the fun of it, the zealots do it because they believe it's better to die after converting under duress. Neither is likely to be following the actual precepts of their religion.
I'm sure you're reporting this from personal experience, as opposed to just repeating what you read on the Onion or Slate, perhaps? Regardless of whether this is true or not - her life's work, the nunneries that still exist around the world, continues to aid the poorest of the poor. If you disagree with their methods, you're always free to create a better organization, are you not?
"Lord of the Flies" anyone?
In addition to the German film, there is another one in the works:
(Written and directed by the writer of 'The Usual Suspects' and 'The Way of the Gun'.)
This shows that some aspects of human behavior are situational and not static. Power and control always influence human interactions.
Categorizing people as good or evil is an oversimplification that takes us further away from understanding human behavior.
TomK's report of Mother Teresa's attitudes toward hospice care have been independently reported by several sources: The Missionary Position: Mother Teresa in Theory and Practice by Christopher Hitchens, Mother Teresa: The Final Verdict by Aroup Chatterjee, and "Mother Theresa's care for the dying" by Robin Fox (Lancet. 1994 Sep 17;344(8925):807-8. [PMID 7818649]).
Among the medical accusations, all confirmed by independent primary sources (people who actually visited her facilities in person): a systematic lack of diagnosis (people often didn't even know what disease they had, or if it was treatable; all diseases were treated as terminal), a systematic lack of analgesia (no pain relief), and frequent reuse of needles (to save money).
There are also many reports of financial irregularities with her charity. Mother Teresa filed a letter with Judge Lance Ito, presiding over the trial of Charles Keating (of the 1980's Savings & Loan scandal), in an attempt to win mercy for one of her major donors. Deputy DA Paul Turley wrote to her in response, asking her to return the money Keating had donated because it had been obtained through defrauding innocent people. She didn't reply.
IMO good works the same way. We're not good or evil by nature, we do *what's expected of us*. Raised in a society that clearly expects us to be considerate of others' rights, we rise to, even exceed, that expectation.
Humans, by nature, obey those we see as authorities, assist and imitate those we see as like ourselves, and hinder and diverge from those we see as unlike ourselves. Realize this, and you are *set free*. (Of course, having freed oneself, the question becomes "what do I do now?")
Here's what I see as the correct answer to the puzzle of the Milgram experiment. Headlock the experimenter. (He *really* isn't expecting that. The peon/master dynamic cuts both ways.) Take his pen, and hold it to his eye. Tell him to order that the subject be released right now, or he never sees in stereo again. When he later claims "but it wasn't real"; tell him, "same goes for me."
To defeat a sociopath you *must* be able to think as they do; *free of any limitations* short of physical capacity. It doesn't mean you have to *do* as they do.
And that's the answer to the question above. Having taken off sheep's clothing, you are not required to be a wolf. There is a third path: the hound. The best white hat hacker wears a black hat underneath. The best sheriff is a gunslinging killer. The best fraud investigator, a con artist. And so on.
Schneier.com is a personal website. Opinions expressed are not necessarily those of BT.