Entries Tagged "bias"

Page 1 of 1

White House Report on Big Data Discrimination

The White House has released a report on big-data discrimination. From the blog post:

Using case studies on credit lending, employment, higher education, and criminal justice, the report we are releasing today illustrates how big data techniques can be used to detect bias and prevent discrimination. It also demonstrates the risks involved, particularly how technologies can deliberately or inadvertently perpetuate, exacerbate, or mask discrimination.

The purpose of the report is not to offer remedies to the issues it raises, but rather to identify these issues and prompt conversation, research­ — and action­ — among technologists, academics, policy makers, and citizens, alike.

The report includes a number of recommendations for advancing work in this nascent field of data and ethics. These include investing in research, broadening and diversifying technical leadership, cross-training, and expanded literacy on data discrimination, bolstering accountability, and creating standards for use within both the government and the private sector. It also calls on computer and data science programs and professionals to promote fairness and opportunity as part of an overall commitment to the responsible and ethical use of data.

Posted on May 6, 2016 at 6:12 AMView Comments

Retelling of Stories Increases Bias

Interesting experiment shows that the retelling of stories increases conflict and bias.

For their study, which featured 196 undergraduates, the researchers created a narrative about a dispute between two groups of young people. It described four specific points of tension, but left purposely ambiguous the issue of which party was the aggressor, and “depicted the groups as equally blameworthy.”

Half of the participants read a version of the story in which the two hostile groups were from two Maryland cities. The other half read a version in which one group was from the city of Gaithersburg, but the other was identified as “your friends.”

Participants were assigned a position between one and four. Those in the first position read the initial version of the story, and then “re-told” it in their own words by writing their version of the events. This was passed on to the person in the second position, who did the same.

The procedure was repeated until all four people had created their own versions of the story. Each new version was then examined for subtle shifts in emphasis, blame, and wording.

The results: Each “partisan communicator” — that is, each student who wrote about the incident involving his or her “friends” — “contributed small distortions that, when accumulated, produced a highly biased, inaccurate representation of the original dispute,” the researchers write.

Standard disclaimer — that American undergraduates might not be the best representatives of our species — applies. But the results are not surprising. We tend to play up the us vs. them narrative when we tell stories. The result is particularly interesting in light of the echo chamber that Internet-based politics has become.

The actual paper is behind a paywall.

EDITED TO ADD (5/14): The paper.

Posted on May 8, 2014 at 7:32 AMView Comments

Cognitive Biases About Violence as a Negotiating Tactic

Interesting paper: Max Abrahms, “The Credibility Paradox: Violence as a Double-Edged Sword in International Politics,” International Studies Quarterly, 2013:

Abstract: Implicit in the rationalist literature on bargaining over the last half-century is the political utility of violence. Given our anarchical international system populated with egoistic actors, violence is thought to promote concessions by lending credibility to their threats. From the vantage of bargaining theory, then, empirical research on terrorism poses a puzzle. For non-state actors, terrorism signals a credible threat in comparison to less extreme tactical alternatives. In recent years, however, a spate of studies across disciplines and methodologies has nonetheless found that neither escalating to terrorism nor with terrorism encourages government concessions. In fact, perpetrating terrorist acts reportedly lowers the likelihood of government compliance, particularly as the civilian casualties rise. The apparent tendency for this extreme form of violence to impede concessions challenges the external validity of bargaining theory, as traditionally understood. In this study, I propose and test an important psychological refinement to the standard rationalist narrative. Via an experiment on a national sample of adults, I find evidence of a newfound cognitive heuristic undermining the coercive logic of escalation enshrined in bargaining theory. Due to this oversight, mainstream bargaining theory overestimates the political utility of violence, particularly as an instrument of coercion.

Posted on October 25, 2013 at 6:30 AMView Comments

Biases in Forensic Science

Some errors in forensic science may be the result of the biases of the examiners:

Though they cannot prove it, Dr Dror and Dr Hampikian suspect the difference in contextual information given to the examiners was the cause of the different results. The original pair may have subliminally interpreted ambiguous information in a way helpful to the prosecution, even though they did not consciously realise what they were doing.


This one example does not prove the existence of a systematic problem. But it does point to a sloppy approach to science. According to Norah Rudin, a forensic-DNA consultant in Mountain View, California, forensic scientists are beginning to accept that cognitive bias exists, but there is still a lot of resistance to the idea, because examiners take the criticism personally and feel they are being accused of doing bad science. According to Dr Rudin, the attitude that cognitive bias can somehow be willed away, by education, training or good intentions, is still pervasive.

Posted on January 31, 2012 at 11:13 AMView Comments

A Critical Essay on the TSA

A critical essay on the TSA from a former assistant police chief:

This is where I find myself now obsessing over TSA policy, or its apparent lack. Every one of us goes to work each day harboring prejudice. This is simply human nature. What I have witnessed in law enforcement over the course of the last two decades serves to remind me how active and passive prejudice can undermine public trust in important institutions, like police agencies. And TSA.

Over the last fifteen years or so, many police agencies started capturing data on police interactions. The primary purpose was to document what had historically been undocumented: informal street contacts. By capturing specific data, we were able to ask ourselves tough questions about potentially biased-policing. Many agencies are still struggling with the answers to those questions.

Regardless, the data permitted us to detect problematic patterns, commonly referred to as passive discrimination. This is a type of discrimination that occurs when we are not aware of how our own biases affect our decisions. This kind of bias must be called to our attention, and there must be accountability to correct it.

One of the most troubling observations I made, at both Albany and BWI, was that — aside from the likely notation in a log (that no one will ever look at) — there was no information captured and I was asked no questions, aside from whether or not I wanted to change my mind.

Given that TSA interacts with tens if not hundreds of millions of travelers each year, it is incredible to me that we, the stewards of homeland security, have failed to insist that data capturing and analysis should occur in a manner similar to what local police agencies have been doing for many years.

EDITED TO ADD (11/12): Follow-on essay by the same person.

Posted on October 29, 2009 at 6:41 AMView Comments

Correspondent Inference Theory

Two people are sitting in a room together: an experimenter and a subject. The experimenter gets up and closes the door, and the room becomes quieter. The subject is likely to believe that the experimenter’s purpose in closing the door was to make the room quieter.

This is an example of correspondent inference theory. People tend to infer the motives — and also the disposition — of someone who performs an action based on the effects of his actions, and not on external or situational factors. If you see someone violently hitting someone else, you assume it’s because he wanted to — and is a violent person — and not because he’s play-acting. If you read about someone getting into a car accident, you assume it’s because he’s a bad driver and not because he was simply unlucky. And — more importantly for this column — if you read about a terrorist, you assume that terrorism is his ultimate goal.

It’s not always this easy, of course. If someone chooses to move to Seattle instead of New York, is it because of the climate, the culture or his career? Edward Jones and Keith Davis, who advanced this theory in the 1960s and 1970s, proposed a theory of “correspondence” to describe the extent to which this effect predominates. When an action has a high correspondence, people tend to infer the motives of the person directly from the action: e.g., hitting someone violently. When the action has a low correspondence, people tend to not to make the assumption: e.g., moving to Seattle.

Like most cognitive biases, correspondent inference theory makes evolutionary sense. In a world of simple actions and base motivations, it’s a good rule of thumb that allows a creature to rapidly infer the motivations of another creature. (He’s attacking me because he wants to kill me.) Even in sentient and social creatures like humans, it makes a lot of sense most of the time. If you see someone violently hitting someone else, it’s reasonable to assume that he’s a violent person. Cognitive biases aren’t bad; they’re sensible rules of thumb.

But like all cognitive biases, correspondent inference theory fails sometimes. And one place it fails pretty spectacularly is in our response to terrorism. Because terrorism often results in the horrific deaths of innocents, we mistakenly infer that the horrific deaths of innocents is the primary motivation of the terrorist, and not the means to a different end.

I found this interesting analysis in a paper by Max Abrahms in International Security. “Why Terrorism Does Not Work” (.PDF) analyzes the political motivations of 28 terrorist groups: the complete list of “foreign terrorist organizations” designated by the U.S. Department of State since 2001. He lists 42 policy objectives of those groups, and found that they only achieved them 7 percent of the time.

According to the data, terrorism is more likely to work if 1) the terrorists attack military targets more often than civilian ones, and 2) if they have minimalist goals like evicting a foreign power from their country or winning control of a piece of territory, rather than maximalist objectives like establishing a new political system in the country or annihilating another nation. But even so, terrorism is a pretty ineffective means of influencing policy.

There’s a lot to quibble about in Abrahms’ methodology, but he seems to be erring on the side of crediting terrorist groups with success. (Hezbollah’s objectives of expelling both peacekeepers and Israel out of Lebanon counts as a success, but so does the “limited success” by the Tamil Tigers of establishing a Tamil state.) Still, he provides good data to support what was until recently common knowledge: Terrorism doesn’t work.

This is all interesting stuff, and I recommend that you read the paper for yourself. But to me, the most insightful part is when Abrahms uses correspondent inference theory to explain why terrorist groups that primarily attack civilians do not achieve their policy goals, even if they are minimalist. Abrahms writes:

The theory posited here is that terrorist groups that target civilians are unable to coerce policy change because terrorism has an extremely high correspondence. Countries believe that their civilian populations are attacked not because the terrorist group is protesting unfavorable external conditions such as territorial occupation or poverty. Rather, target countries infer the short-term consequences of terrorism — the deaths of innocent civilians, mass fear, loss of confidence in the government to offer protection, economic contraction, and the inevitable erosion of civil liberties — (are) the objects of the terrorist groups. In short, target countries view the negative consequences of terrorist attacks on their societies and political systems as evidence that the terrorists want them destroyed. Target countries are understandably skeptical that making concessions will placate terrorist groups believed to be motivated by these maximalist objectives.

In other words, terrorism doesn’t work, because it makes people less likely to acquiesce to the terrorists’ demands, no matter how limited they might be. The reaction to terrorism has an effect completely opposite to what the terrorists want; people simply don’t believe those limited demands are the actual demands.

This theory explains, with a clarity I have never seen before, why so many people make the bizarre claim that al Qaeda terrorism — or Islamic terrorism in general — is “different”: that while other terrorist groups might have policy objectives, al Qaeda’s primary motivation is to kill us all. This is something we have heard from President Bush again and again — Abrahms has a page of examples in the paper — and is a rhetorical staple in the debate. (You can see a lot of it in the comments to this previous essay.)

In fact, Bin Laden’s policy objectives have been surprisingly consistent. Abrahms lists four; here are six from former CIA analyst Michael Scheuer’s book Imperial Hubris:

  1. End U.S. support of Israel
  2. Force American troops out of the Middle East, particularly Saudi Arabia
  3. End the U.S. occupation of Afghanistan and (subsequently) Iraq
  4. End U.S. support of other countries’ anti-Muslim policies
  5. End U.S. pressure on Arab oil companies to keep prices low
  6. End U.S. support for “illegitimate” (i.e. moderate) Arab governments, like Pakistan

Although Bin Laden has complained that Americans have completely misunderstood the reason behind the 9/11 attacks, correspondent inference theory postulates that he’s not going to convince people. Terrorism, and 9/11 in particular, has such a high correspondence that people use the effects of the attacks to infer the terrorists’ motives. In other words, since Bin Laden caused the death of a couple of thousand people in the 9/11 attacks, people assume that must have been his actual goal, and he’s just giving lip service to what he claims are his goals. Even Bin Laden’s actual objectives are ignored as people focus on the deaths, the destruction and the economic impact.

Perversely, Bush’s misinterpretation of terrorists’ motives actually helps prevent them from achieving their goals.

None of this is meant to either excuse or justify terrorism. In fact, it does the exact opposite, by demonstrating why terrorism doesn’t work as a tool of persuasion and policy change. But we’re more effective at fighting terrorism if we understand that it is a means to an end and not an end in itself; it requires us to understand the true motivations of the terrorists and not just their particular tactics. And the more our own cognitive biases cloud that understanding, the more we mischaracterize the threat and make bad security trade-offs.

This is my 46th essay for Wired.com, based on a paper I blogged about last week (there are a lot of good comments to that blog post).

Posted on July 12, 2007 at 12:59 PMView Comments

Sidebar photo of Bruce Schneier by Joe MacInnis.