Schneier on Security
A blog covering security and security technology.
« The Problem of False Alarms |
| Bar Code Switching »
May 30, 2012
The Psychology of Immoral (and Illegal) Behavior
When I talk about Liars and Outliers to security audiences, one of the things I stress is our traditional security focus -- on technical countermeasures -- is much narrower than it could be. Leveraging moral, reputational, and institutional pressures are likely to be much more effective in motivating cooperative behavior.
This story illustrates the point. It's about the psychology of fraud, "why good people do bad things."
There is, she says, a common misperception that at moments like this, when people face an ethical decision, they clearly understand the choice that they are making.
"We assume that they can see the ethics and are consciously choosing not to behave ethically," Tenbrunsel says.
This, generally speaking, is the basis of our disapproval: They knew. They chose to do wrong.
But Tenbrunsel says that we are frequently blind to the ethics of a situation.
Over the past couple of decades, psychologists have documented many different ways that our minds fail to see what is directly in front of us. They've come up with a concept called "bounded ethicality": That's the notion that cognitively, our ability to behave ethically is seriously limited, because we don't always see the ethical big picture.
One small example: the way a decision is framed. "The way that a decision is presented to me," says Tenbrunsel, "very much changes the way in which I view that decision, and then eventually, the decision it is that I reach."
Essentially, Tenbrunsel argues, certain cognitive frames make us blind to the fact that we are confronting an ethical problem at all.
Tenbrunsel told us about a recent experiment that illustrates the problem. She got together two groups of people and told one to think about a business decision. The other group was instructed to think about an ethical decision. Those asked to consider a business decision generated one mental checklist; those asked to think of an ethical decision generated a different mental checklist.
Tenbrunsel next had her subjects do an unrelated task to distract them. Then she presented them with an opportunity to cheat.
Those cognitively primed to think about business behaved radically different from those who were not -- no matter who they were, or what their moral upbringing had been.
"If you're thinking about a business decision, you are significantly more likely to lie than if you were thinking from an ethical frame," Tenbrunsel says.
According to Tenbrunsel, the business frame cognitively activates one set of goals -- to be competent, to be successful; the ethics frame triggers other goals. And once you're in, say, a business frame, you become really focused on meeting those goals, and other goals can completely fade from view.
Typically when we hear about large frauds, we assume the perpetrators were driven by financial incentives. But psychologists and economists say financial incentives don't fully explain it. They're interested in another possible explanation: Human beings commit fraud because human beings like each other.
We like to help each other, especially people we identify with. And when we are helping people, we really don't see what we are doing as unethical.
The article even has some concrete security ideas:
Now if these psychologists and economists are right, if we are all capable of behaving profoundly unethically without realizing it, then our workplaces and regulations are poorly organized. They're not designed to take into account the cognitively flawed human beings that we are. They don't attempt to structure things around our weaknesses.
Some concrete proposals to do that are on the table. For example, we know that auditors develop relationships with clients after years of working together, and we know that those relationships can corrupt their audits without them even realizing it. So there is a proposal to force businesses to switch auditors every couple of years to address that problem.
Another suggestion: A sentence should be placed at the beginning of every business contract that explicitly says that lying on this contract is unethical and illegal, because that kind of statement would get people into the proper cognitive frame.
Along similar lines, some years ago Ross Anderson made the suggestion that the webpages of peoples' online bank accounts should include their photographs, based on the research that it's harder to commit fraud against someone who you identify with as a person.
Two excellent papers on this topic:
- Nina Mazar and Dan Ariely, "Dishonesty in Everyday Life and its Policy Implications," Journal of Public Policy and Marketing, 2006, vol. 25, No. 1: 117-126.
- Nina Mazar, On Amir, and Dan Ariely, "The Dishonesty of Honest People: A Theory of Self-Concept Maintenance," Journal of Marketing Research, 2008, vol. 45: 633-634.
Abstract of the second paper:
Dishonesty plays a large role in the economy. Causes for (dis)honest behavior seem to be based partially on external rewards, and partially on internal rewards. Here, we investigate how such external and internal rewards work in concert to produce (dis)honesty. We propose and test a theory of self-concept maintenance that allows people to engage to some level in dishonest behavior, thereby benefiting from external benefits of dishonesty, while maintaining their positive view about themselves in terms of being honest individuals. The results show that (1) given the opportunity to engage in beneficial dishonesty, people will engage in such behaviors; (2) the amount of dishonesty is largely insensitive to either the expected external benefits or the costs associated with the deceptive acts; (3) people know about their actions but do not update their self-concepts; (4) causing people to become more aware of their internal standards for honesty decreases their tendency for deception; and (5) increasing the "degrees of freedom" that people have to interpret their actions increases their tendency for deception. We suggest that dishonesty governed by self-concept maintenance is likely to be prevalent in the economy, and understanding it has important implications for designing effective methods to curb dishonesty.
Posted on May 30, 2012 at 12:54 PM
• 34 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
About the photograph on online bank accounts...
I guess, much online banking fraud is done using bots nowadays. They are immune against that kind of priming. And identity thefts cleaning an account might even have other use for such a photograph.
While I can see the merits in such fresh approaches against wrongdoing, there are probably a lot of possibilities to get it wrong...
The problem with these type of studies is that they assume dishonesty exists and even if it does exist that curbing it is a "good thing". But lets be honest; the overwhelming majority of words both written and spoken are lies, certainly all the greatest works in literature are lies. But we perceive some lies as "good lies" or "white lies" and other lies as bad lies.
Most of the research on "honesty" is Orwellian in nature: "my lies good your lies bad". It doesn't advance the conversation though it sure is flattering to the powers that be.
I read "Toby's story" in the article, and I see nothing that indicates that he, and the people who helped him, didn't recognize what they were doing as wrong or unethical; they simply thought there was a higher need of some kind. Moral/ethical decisions, even ordinary everyday ones, often involve not good and bad but bad and worse. The complicating factor is that while "good" and "bad" might be generally agreed on, "bad" and "worse" tend to be highly subjective.
This may very well apply to the initial decision to "do wrong," but the primary psychological factor after I've already committed some level of fraud or illegal behavior is cognitive dissonance and a rationalization of my existing behavior as appropriate and ethical.
Once you have someone at that point, things like framing are far less effective.
Also, I would argue that self-preservation is an even stronger motivator in preventing white-collar illegal activities than ethics. I wager it'd be far more effective to start off contracts or classified material with a picture of a large and angry prison inmate and the caption "If you lie or leak this material, Bubba will want you to bend over."
Business is not the only frame that may corrupt ethical perception, Government is another major source of ethical blindness. The idea that people don't recognise when they are not behaving morally certainly explains how the authors don't see how "[forcing] businesses to switch auditors every couple of years to address that problem" is immoral. It may also be an indication that moral priming didn't help the authors in this matter.
A business, just by having the same auditor for many years, has not crossed any moral or ethical line, yet under this proposal the unprovoked force of government is used against them if they don't comply in changing their auditor.
Had the authors of this article not been wearing their government issue ethics blinders they might have instead persuaded businesses using arguments to take this measure up voluntarily as good practice to prevent cosy relationships with auditors forming.
Yes, honesty in any business needs little bit of secrecy of the trade too. But the survival of the fittest factor makes people tell lies. Here is a very nice thing Mahatma Gandhi has said in the same context:
"..its difficult but not impossible to conduct strictly honest business.."
"Some concrete proposals to do that are on the table. For example, we know that auditors develop relationships with clients after years of working together, and we know that those relationships can corrupt their audits without them even realizing it. So there is a proposal to force businesses to switch auditors every couple of years to address that problem."
In Australia, I believe listed companies are required to switch audit partners every 3 years and audit companies every 5 years.
Forcing companies to change auditors is immoral? Seriously? These laws have been introduced because of real world, serious failures in the audit process, with multi-millions of shareholders monies lost to fraud, and a lax or complacent audit process failing to pick up the obvious signs.
"Another suggestion: A sentence should be placed at the beginning of every business contract that explicitly says that lying on this contract is unethical and illegal, because that kind of statement would get people into the proper cognitive frame."
I saw some kind of economist on TV last night, who said that making people put their name, signature and declaration at the top of an insurance claim form rather than at the end dramatically reduces the amount of fraud attempts.
This recent research does not jibe with dozens of previous studies. In those studies, people were placed in situations where they could commit an illegal or immoral act and believed they had no chance of being detected. Many (sometimes the majority) chose to commit the act. When questioned afterwards, ALL of them admitted they knowingly made an illegal and/or immoral choice, but the combination of greed and the belief that they would be undetected overrode the moral and legal aspects. Thus, I do not buy into the claim that people who commit fraud do not know they are doing so. For example, does anyone believe that the telemarketing staff for a Ponzi scheme have no idea that they are abetting a crime?
Other studies show that almost two-thirds of college students cheat on exams and/or term paper writing. Every one of those students knows that cheating is unethical and a violation of college rules, but the benefits of cheating are high enough and the chances of being caught are low enough to override what little moral qualms they may have.
I believe that most people behave unethically whenever they believe the combination of reward size, detection risk, and penalty-if-detected size favors the unethical or illegal behavior. This applies to all kinds of behaviors: cheating on spouses, insurance fraud, working under-the-table, government benefit fraud, buying stolen goods, using banned recreational drugs, etc.
I don't see a contradiction, necessarily. There are varying degrees of awareness.
Priming someone in an ethical mindset may plausibly cause them to be more *acutely* aware of the moral implications of their decisions, and that heightened awareness could plausibly lead to changes in behavior, even if they would not have been completely oblivious otherwise.
Also, I'm not sure that self-reporting on your mindset after the fact is necessarily reliable.
Actually, if using the same auditor for a prolonged period of time significantly increases the risk malfeasance and the cost of switching is negligible, I think you could make a very compelling argument that failure to switch is immoral.
Just like if you fail to follow any other safety rule, it won't *necessarily* result in anything bad happening, but you are recklessly endangering the public by your negligence.
I believe you are the first person I have ever seen suggest that laws requiring reasonable and prudent behavior would be unethical. Do you also consider it immoral that we require people to pass a driving test and obtain a license before driving cars on public roads?
"I believe you are the first person I have ever seen suggest that laws requiring reasonable and prudent behavior would be unethical."
Such a law wouldn't be a "requirement", it's a gun against your head that says "You Will Change Your Auditors!". Skipping passed that moral problem is not solved by the perceived benefits, nor by how trivial you perceive the cost to be.
@Someone: "... Priming someone in an ethical mindset may plausibly cause them to be more *acutely* aware of the moral implications of their decisions..."
It may be plausible, but the evidence does not support it. Going back to the college example, students are reminded often that using term papers written by others and cheating on exams are counterproductive (the students won't learn as much), unethical, and can result in expulsion. Some colleges have students sign pledges that they will not cheat. Yet study after study confirms that over 60% of college students cheat at least once. (I was disappointed to learn that premedical students are just as likely to cheat as others.)
I believe that Tenbrunsels experiment scenario was unrealistic and fails to model what happens in the real world. It looks like yet another instance where the authors formed their conclusions and then gathered data.
Wow, poor Toby. I don't know if it's worse that he broke his promise with his father when he was dead or alive (I guess only people who had relationships with their fathers would understand)...
To me it's pretty simple, and I know this isn't a novel concept but: desperation=fraud-causing behavior. It's economics, "it's the economy, stupid". "If I have to sell crack cocaine to feed me and my family, then I'll do it." "If I must commit fraud to not be the drive-thru guy at McDonalds or the greeter at Wal-mart, then fraud it is!" Isn't it obvious to everyone? As another bazillionaire is created (yes, bazillionaire) and another 1000 homeless people are created, will this rise (that is already apparent to me at least) in unethical behavior be a surprise? So if wealthy people (multi-billionaires) ever had an incentive to "reinvest" their extraordinary riches, or at the very least stop being "money-grubbing cockroaches", it's that they may well see the apocalyptic downfall of society that we can all feel on the horizon but hope will never come. If you want to be realist, I guess just think about what happens when you have more people (~7 billion & counting) competing for the same amount of resources.
One of my ideas to reduce hunger-desperation is to take advantage of all the open land that could be used for crops (food) like all the space around suburban homes. One of my dreams is to make the fear of starvation a thing of the past, there is so much unused space to grow food! I just made a trip to a farm "down south" to plant some corn and watermelon, I'm also planting tomatoes, cucumbers, green peppers, sunflower seeds, and peas; but I want to expand my operation and get more people involved. It's a magical feeling to see a seedling begin to sprout, it's a life-form you started...
You seem to be saying that all laws or a subset of laws are immoral. How do YOU conclude that a law OF THIS TYPE (forcing companies to change Auditors periodically, with the intention of reducing "irregularities", however unconcious such irregularities may be) is "immoral".
Is it all laws that are immoral? Laws which penalise certain behaviour (and if so, why is this wrong)? Laws which affect business (and why should business be immune to laws)? What is the objection here?
Your view on [im]morality of law seems to have a huge assumption at its core, which you have not articulated, and which I am at a loss to understand.
I know people who, once they cross a big ethical boundary, find it easier to cross it again and again. Their sense of right and wrong may have always been a little fuzzy. Perhaps do to childhood trauma or because one parent was moral while the other was not. That can lead to an almost passive/aggressive kind of morality. It seems like these folks, if surrounded by moral people, stay on the honesty path. But, once they are on their own or worse yet, influenced by other's unethical behavior, they go down the opposite path. Although they seem to continue to see themselves as the person they were before the change and are somehow incredibly good at not only justifying their behavior but actually viewing their victims as the cause of the problem. Seems to me that with the current social computer networks, dishonesty has grown by leaps and bounds, along with passive/aggression morality and the inability to read body language and facial cues. A very important talent when trying to decide if a person is being honest.
Another part of the "business" frame that can encourage unethical behavior is the sense that one's counterparts are also likely to be behaving unethically. People can justify taking from their employer on the grounds that they're underpaid or otherwise cheated, from a store based on the grounds that it overcharges, and so forth.
There used to be a perception that it was less unethical to cheat large, faceless corporations than to cheat mom-and-pop stores; I wonder if that's still true.
This certainly sounds like a plausible explanation for how people commit opportunistic fraud or get caught up in large-scale fraudulent schemes, but I think we should also keep in mind that some people are just psychopaths.
A lot of the people that Toby pulled into his fraud probably got involved because of the factors described in the article, but Toby himself sounds like a psychopath. "I'm not fixing anything," rather than "I'm stealing from other people." The substance abuse. The practiced ease with which he invoked sympathy in other people to get them to lie for him. Sure, he sound contrite now, but isn't that what people want to hear?
'Although they seem to continue to see themselves as the person they were before the change and are somehow incredibly good at not only justifying their behavior but actually viewing their victims as the cause of the problem'
Such behaviour is described by 'cognitive dissonance'
Actually, it is all about greed/lust. Greed for money and lust for power. In religion and politics, it is about power (control). In business and government, it is about money. The finish line is always about having neither though the pretenders believe it is about having both.
So, here's another article describing some studies on why we lie and the impact of moral reminders. I think that, as humans, we have a particularly difficult time understanding human nature, and while some things seem like common sense, we should back them up with work such as this: http://online.wsj.com/article/...
I've also written a little about how we can take this knowledge and start to apply it to our risk management techniques, essentially arguing that consideration of human nature needs to be embedded not only in policy, process, and procedure, but also in the tools we use along that workflow. http://www.tripwire.com/state-of-security/...
@Antonio Lorusso: "Such a law wouldn't be a 'requirement', it's a gun against your head"
All laws are ultimately enforced by (threat of) violence, so similar statements could be made about them all. Unless you are arguing that all possible laws are inherently immoral, I don't see how this supports your position.
Signing a pledge not to cheat at your time of enrollment is not "priming your mindset" for a moral decision you make 6 months later (at least not in the sense used in this study), so I fail to see how any of the evidence you cited is relevant.
But even if the subjects were primed, the fact that a lot of them cheated would still not disprove the theory that priming reduces cheating. You'd need to have a control group and compare the rate of cheating between them.
And a 60% rate of cheating "at least once" doesn't even sound terribly high. The number of separate opportunities to cheat over the course of an entire college career must number in the thousands, at least, so if cheating were distributed randomly, that would indicate less than a 0.1% rate of cheating at any given opportunity. (Not that I'm suggesting each decision to cheat is an independent random event, just trying to illustrate a point.)
A sentence should be placed at the beginning of every business contract that explicitly says that lying on this contract is unethical and illegal, ... .
If each contract would start with the same sentence, people would learn to ignore. It would became an empty annoying ritual, just like do not copy messages in the DVDs.
Am I correct in my understanding then, that both of you think that holding a Gun to someone's head to force them to change their Auditor (which is what this law and it's enforcement ultimately entails) is moral? That there isn't a single moral principle you hold that this breaks?
Context is important. If some random guy with a gun walk up to you and demands that you install a smoke detector in your business, I can see several problems with that. If the government passes a law saying that all businesses must have fire safety systems meeting certain well-defined guidelines, and then uses its law enforcement arm to arrest, try, and punish the owners of businesses that don't (following due process), that's a completely different story.
Once again: the only objection you have so far raised to this particular law is an objection that can be raised with equally validity against every possible law that could ever exist.
If you honestly feel that's a problem, then you don't have any problem with an auditor-changing law specifically, you have a problem with the whole concept of having laws in the first place. That's a separate discussion not really related to this news post, and your apparent position on that topic puts you solidly in the minority camp.
But if you think that having laws might possibly be OK under some conceivable circumstances, then you have yet to suggest any reason that this specific one would be immoral.
All laws, as someone mentioned, are based on force. An acquaintance of mine had a parking ticket, which he failed to pay. When he was stopped the old ticket came up and he failed to produce a proof of insurance. Then he failed to appear in court for these matters. Eventually, a judge put out a bench warrant for him and guaranteed at least two weeks in the county lockup. So, yes, eventually "men with guns" enforce even parking meters...
@Escalator & Antonio... Force need not require guns - fines can in principle be enforced by attachment of earnings or confiscation of assets (through the banking system, for example). So unless you're some kind of off-grid bunker-dweller, legal compulsion need not "ultimately" require lethal force. Even if police are involved, not all jursidictions use armed police.
Meanwhile, back at the topic of motivating cooperative behaviour, it seems to me that different societies have different levels of "acceptable" behaviour. In Europe, for instance, it is quite acceptable for society to cooperate to provide, say, health care for its members. In the USA, not so much. And surveys show that in China, increasing the cost of umbrellas when it is raining is viewed as more acceptable that it would be in Germany.
Such base levels of cooperativeness may well affect how willing people are to see or consider their choices as having an ethical dimension. Or maybe it is the other way round.
Police Officer is the title of a paid job, not a moral exemption. They are a person, no different from you and me, and thus must be held to the same morality, as you and I.
If it is immoral for someone called "Random Person Bob" to hold a gun to someone's head to get them to change their Auditor it must also be immoral for someone called "Officer Bob" to do the same.
This is an article on ethical blinders, the inability to see what are doing (or proposing) is immoral. I am proposing that the justifications and distractions (just a few: 1. arguments from popularity. 2. repeated appeals to law when I have been discussing a specifc moral proposition) you are using to skip past the moral properties of this specific proposition, is an ethical blinder, because you have already acknowledged that a "random guy" who did this would be immoral, and are using rationalisations of a job title, a uniform, and argument from popularity to make something that would be immoral and unacceptable if you or I did it, suddenly moral.
All laws are ultimately enforced by guns. That most people (sensibly) don't play chicken with the guns, doesn't change that fact, or the moral question I am asking.
Your idea of "motivating co-operative behaviour" is to threaten people with guns to FORCE them to do it your way? Feel free to never invite me to any team building seminar you may ever hold.
Consider the following examples:
1) A court of law examines public records and determines that you did not switch auditors according to the schedule prescribed by law, and so fines you $100, all in due accordance with the laws of the land
2) A police officer uses radar to determine that you are driving in excess of posted speed limits, pulls you over, and writes you a ticket requiring you to pay a fine of $100, all in due accordance with the laws of the land
If you can identify a compelling reason to consider #1 immoral but still consider #2 to be moral, I would be interested in hearing it.
3) A person walks up to you on the street, pulls a gun on you, and demands that you pay him $100
Personally, I see a very great moral difference between #2 and #3. This difference is related to impartiality, advance notice, implied consent, and procedures designed to minimize errors. It has nothing to do with job titles, uniforms, or popularity. I'm reasonably certain that my position on this matter is very nearly universal.
I'm not saying that you have to agree with me, but if you disagree, you should be aware that most people you talk to will likely require some explanation of your moral philosophy before they can even follow your arguments, let alone agree with them.
All fines are backed by guns, so that point is irrelevant.
Since you agree with me that in the case of a "random person" using force to get someone to change their auditor is immoral, you by definition are clearly able to follow and understand my moral argument.
What you haven't explained yet explained is why that moral proposition changes when a person called "Officer Bob" is paid to force someone to change their auditor.
4) The government requires that everytime you have sex, you are required to make a $100 donation to the Republican Party.
Would this be ethical? Would it be different if it were the Democratic Party? I think that the key to this argument is the orginal stipulation 'reasonable'. Unfortunatly reasonable is a very reasonable stipulation for a law and just as unfortunatly it is un-definable. I doubt many changed their minds when the receiver of the donation changed from the Republican to the Democratic Party (come on, fess up) but does that pass the reasonable test. How 'bout the Catholic Church? Or the church of your choice?
Or is the whole idea of taxing sex unreasonable.
Was there, once upon a time, a country founded on this idea (that not every law was ethical)?
Do you have a reference for the Ross Anderson quote about photos on bank pages? I have been thinking along similar lines for minimizing "friendly fraud" with mobile devices. I would like to make sure I'm not missing important earlier work. Thanks!
Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc.