Page 344

IT Security and the Normalization of Deviance

Professional pilot Ron Rapp has written a fascinating article on a 2014 Gulfstream plane that crashed on takeoff. The accident was 100% human error and entirely preventable—the pilots ignored procedures and checklists and warning signs again and again. Rapp uses it as example of what systems theorists call the “normalization of deviance,” a term coined by sociologist Diane Vaughan:

Social normalization of deviance means that people within the organization become so much accustomed to a deviant behaviour that they don’t consider it as deviant, despite the fact that they far exceed their own rules for the elementary safety. But it is a complex process with some kind of organizational acceptance. The people outside see the situation as deviant whereas the people inside get accustomed to it and do not. The more they do it, the more they get accustomed. For instance in the Challenger case there were design flaws in the famous “O-rings,” although they considered that by design the O-rings would not be damaged. In fact it happened that they suffered some recurrent damage. The first time the O-rings were damaged the engineers found a solution and decided the space transportation system to be flying with “acceptable risk.” The second time damage occurred, they thought the trouble came from something else. Because in their mind they believed they fixed the newest trouble, they again defined it as an acceptable risk and just kept monitoring the problem. And as they recurrently observed the problem with no consequence they got to the point that flying with the flaw was normal and acceptable. Of course, after the accident, they were shocked and horrified as they saw what they had done.

The point is that normalization of deviance is a gradual process that leads to a situation where unacceptable practices or standards become acceptable, and flagrant violations of procedure become normal—despite that fact that everyone involved knows better.

I think this is a useful term for IT security professionals. I have long said that the fundamental problems in computer security are not about technology; instead, they’re about using technology. We have lots of technical tools at our disposal, and if technology alone could secure networks we’d all be in great shape. But, of course, it can’t. Security is fundamentally a human problem, and there are people involved in security every step of the way. We know that people are regularly the weakest link. We have trouble getting people to follow good security practices and not undermine them as soon as they’re inconvenient. Rules are ignored.

As long as the organizational culture turns a blind eye to these practices, the predictable result is insecurity.

None of this is unique to IT. Looking at the healthcare field, John Banja identifies seven factors
that contribute to the normalization of deviance:

  • The rules are stupid and inefficient!
  • Knowledge is imperfect and uneven.
  • The work itself, along with new technology, can disrupt work behaviors and rule compliance.
  • I’m breaking the rule for the good of my patient!
  • The rules don’t apply to me/you can trust me.
  • Workers are afraid to speak up.
  • Leadership withholding or diluting findings on system problems.

Dan Luu has written about this, too.

I see these same factors again and again in IT, especially in large organizations. We constantly battle this culture, and we’re regularly cleaning up the aftermath of people getting things wrong. The culture of IT relies on single expert individuals, with all the problems that come along with that. And false positives can wear down a team’s diligence, bringing about complacency.

I don’t have any magic solutions here. Banja’s suggestions are good, but general:

  • Pay attention to weak signals.
  • Resist the urge to be unreasonably optimistic.
  • Teach employees how to conduct emotionally uncomfortable conversations.
  • System operators need to feel safe in speaking up.
  • Realize that oversight and monitoring are never-ending.

The normalization of deviance is something we have to face, especially in areas like incident response where we can’t get people out of the loop. People believe they know better and deliberately ignore procedure, and invariably forget things. Recognizing the problem is the first step toward solving it.

This essay previously appeared on the Resilient Systems blog.

Tags: , , , ,

Posted on January 11, 2016 at 6:45 AM • View Comments

"How Stories Deceive"

Fascinating New Yorker article about Samantha Azzopardi, serial con artist and deceiver.

The article is really about how our brains allow stories to deceive us:

Stories bring us together. We can talk about them and bond over them. They are shared knowledge, shared legend, and shared history; often, they shape our shared future. Stories are so natural that we don’t notice how much they permeate our lives. And stories are on our side: they are meant to delight us, not deceive us—an ever-present form of entertainment.

That’s precisely why they can be such a powerful tool of deception. When we’re immersed in a story, we let down our guard. We focus in a way we wouldn’t if someone were just trying to catch us with a random phrase or picture or interaction. (“He has a secret” makes for a far more intriguing proposition than “He has a bicycle.”) In those moments of fully immersed attention, we may absorb things, under the radar, that would normally pass us by or put us on high alert. Later, we may find ourselves thinking that some idea or concept is coming from our own brilliant, fertile minds, when, in reality, it was planted there by the story we just heard or read.

Tags: , , ,

Posted on January 8, 2016 at 12:54 PM • View Comments

Replacing Judgment with Algorithms

China is considering a new “social credit” system, designed to rate everyone’s trustworthiness. Many fear that it will become a tool of social control—but in reality it has a lot in common with the algorithms and systems that score and classify us all every day.

Human judgment is being replaced by automatic algorithms, and that brings with it both enormous benefits and risks. The technology is enabling a new form of social control, sometimes deliberately and sometimes as a side effect. And as the Internet of Things ushers in an era of more sensors and more data—and more algorithms—we need to ensure that we reap the benefits while avoiding the harms.

Right now, the Chinese government is watching how companies use “social credit” scores in state-approved pilot projects. The most prominent one is Sesame Credit, and it’s much more than a financial scoring system.

Citizens are judged not only by conventional financial criteria, but by their actions and associations. Rumors abound about how this system works. Various news sites are speculating that your score will go up if you share a link from a state-sponsored news agency and go down if you post pictures of Tiananmen Square. Similarly, your score will go up if you purchase local agricultural products and down if you purchase Japanese anime. Right now the worst fears seem overblown, but could certainly come to pass in the future.

This story has spread because it’s just the sort of behavior you’d expect from the authoritarian government in China. But there’s little about the scoring systems used by Sesame Credit that’s unique to China. All of us are being categorized and judged by similar algorithms, both by companies and by governments. While the aim of these systems might not be social control, it’s often the byproduct. And if we’re not careful, the creepy results we imagine for the Chinese will be our lot as well.

Sesame Credit is largely based on a US system called FICO. That’s the system that determines your credit score. You actually have a few dozen different ones, and they determine whether you can get a mortgage, car loan or credit card, and what sorts of interest rates you’re offered. The exact algorithm is secret, but we know in general what goes into a FICO score: how much debt you have, how good you’ve been at repaying your debt, how long your credit history is and so on.

There’s nothing about your social network, but that might change. In August, Facebook was awarded a patent on using a borrower’s social network to help determine if he or she is a good credit risk. Basically, your creditworthiness becomes dependent on the creditworthiness of your friends. Associate with deadbeats, and you’re more likely to be judged as one.

Your associations can be used to judge you in other ways as well. It’s now common for employers to use social media sites to screen job applicants. This manual process is increasingly being outsourced and automated; companies like Social Intelligence, Evolv and First Advantage automatically process your social networking activity and provide hiring recommendations for employers. The dangers of this type of system—from discriminatory biases resulting from the data to an obsession with scores over more social measures—are too many.

The company Klout tried to make a business of measuring your online influence, hoping its proprietary system would become an industry standard used for things like hiring and giving out free product samples.

The US government is judging you as well. Your social media postings could get you on the terrorist watch list, affecting your ability to fly on an airplane and even get a job. In 2012, a British tourist’s tweet caused the US to deny him entry into the country. We know that the National Security Agency uses complex computer algorithms to sift through the Internet data it collects on both Americans and foreigners.

All of these systems, from Sesame Credit to the NSA’s secret algorithms, are made possible by computers and data. A couple of generations ago, you would apply for a home mortgage at a bank that knew you, and a bank manager would make a determination of your creditworthiness. Yes, the system was prone to all sorts of abuses, ranging from discrimination to an old-boy network of friends helping friends. But the system also couldn’t scale. It made no sense for a bank across the state to give you a loan, because they didn’t know you. Loans stayed local.

FICO scores changed that. Now, a computer crunches your credit history and produces a number. And you can take that number to any mortgage lender in the country. They don’t need to know you; your score is all they need to decide whether you’re trustworthy.

This score enabled the home mortgage, car loan, credit card and other lending industries to explode, but it brought with it other problems. People who don’t conform to the financial norm—having and using credit cards, for example—can have trouble getting loans when they need them. The automatic nature of the system enforces conformity.

The secrecy of the algorithms further pushes people toward conformity. If you are worried that the US government will classify you as a potential terrorist, you’re less likely to friend Muslims on Facebook. If you know that your Sesame Credit score is partly based on your not buying “subversive” products or being friends with dissidents, you’re more likely to overcompensate by not buying anything but the most innocuous books or corresponding with the most boring people.

Uber is an example of how this works. Passengers rate drivers and drivers rate passengers; both risk getting booted out of the system if their rankings get too low. This weeds out bad drivers and passengers, but also results in marginal people being blocked from the system, and everyone else trying to not make any special requests, avoid controversial conversation topics, and generally behave like good corporate citizens.

Many have documented a chilling effect among American Muslims, with them avoiding certain discussion topics lest they be taken the wrong way. Even if nothing would happen because of it, their free speech has been curtailed because of the secrecy surrounding government surveillance. How many of you are reluctant to Google “pressure cooker bomb”? How many are a bit worried that I used it in this essay?

This is what social control looks like in the Internet age. The Cold-War-era methods of undercover agents, informants living in your neighborhood, and agents provocateur is too labor-intensive and inefficient. These automatic algorithms make possible a wholly new way to enforce conformity. And by accepting algorithmic classification into our lives, we’re paving the way for the same sort of thing China plans to put into place.

It doesn’t have to be this way. We can get the benefits of automatic algorithmic systems while avoiding the dangers. It’s not even hard.

The first step is to make these algorithms public. Companies and governments both balk at this, fearing that people will deliberately try to game them, but the alternative is much worse.

The second step is for these systems to be subject to oversight and accountability. It’s already illegal for these algorithms to have discriminatory outcomes, even if they’re not deliberately designed in. This concept needs to be expanded. We as a society need to understand what we expect out of the algorithms that automatically judge us and ensure that those expectations are met.

We also need to provide manual systems for people to challenge their classifications. Automatic algorithms are going to make mistakes, whether it’s by giving us bad credit scores or flagging us as terrorists. We need the ability to clear our names if this happens, through a process that restores human judgment.

Sesame Credit sounds like a dystopia because we can easily imagine how the Chinese government can use a system like this to enforce conformity and stifle dissent. Our own systems seem safer, because we don’t believe the corporations and governments that run them are malevolent. But the dangers are inherent in the technologies. As we move into a world where we are increasingly judged by algorithms, we need to ensure that they do so fairly and properly.

This essay previously appeared on CNN.com.

Tags: , , , , , ,

Posted on January 8, 2016 at 5:21 AM • View Comments

Straight Talk about Terrorism

Nice essay that lists ten “truths” about terrorism:

  1. We can’t keep the bad guys out.
  2. Besides, the threat is already inside.
  3. More surveillance won’t get rid of terrorism, either.
  4. Defeating the Islamic State won’t make terrorism go away.
  5. Terrorism still remains a relatively minor threat, statistically speaking.
  6. But don’t relax too much, because things will probably get worse before they get better.
  7. Meanwhile, poorly planned Western actions can make things still worse.
  8. Terrorism is a problem to be managed.
  9. To do this, however, we need to move beyond the political posturing that characterizes most public debates about counterterrorism and instead speak honestly about the costs and benefits of different approaches.
  10. We need to stop rewarding terrorism.

Nothing here will be news to regular readers of this blog.

Tags: ,

Posted on January 7, 2016 at 7:00 AM • View Comments

How the US Is Playing Both Ends on Data Privacy

There’s an excellent article in Foreign Affairs on how the European insistence on data privacy—most recently illustrated by their invalidation of the “safe harbor” agreement—is really about the US talking out of both sides of its mouth on the issue: championing privacy in public, but spying on everyone in private. As long as the US keeps this up, the authors argue, this issue will get worse.

From the conclusion:

The United States faces a profound choice. It can continue to work in a world of blurred lines and unilateral demands, making no concessions on surveillance and denouncing privacy rights as protectionism in disguise. Yet if it does so, it is U.S. companies that will suffer.

Alternatively, it can recognize that globalization comes in different flavors and that Europeans have real and legitimate problems with ubiquitous U.S. surveillance and unilateralism. An ambitious strategy would seek to reform EU and U.S. privacy rules so as to put in place a comprehensive institutional infrastructure that could protect the privacy rights of European and U.S. citizens alike, creating rules and institutions to restrict general surveillance to uses that are genuinely in the security interests of all the countries.

More broadly, the United States needs to disentangle the power of a U.S.-led order from the temptations of manipulating that order to its national security advantage. If it wants globalization to continue working as it has in the past, the United States is going to have to stop thinking of flows of goods and information as weapons and start seeing them as public goods that need to be maintained and nurtured. Ultimately, it is U.S. firms and the American economy that stand to benefit most.

EDITED TO ADD (1/13): Stewart Baker on the same topic.

Tags: , , , ,

Posted on January 6, 2016 at 6:14 AM • View Comments

NSA Spies on Israeli Prime Minister

The Wall Street Journal has a story that the NSA spied on Israeli Prime Minister Benjamin Netanyahu and other Israeli government officials, and incidentally collected conversations between US citizens—including lawmakers—and those officials.

US lawmakers who are usually completely fine with NSA surveillance are aghast at this behavior, as both Glenn Greenwald and Trevor Timm explain. Greenwald:

So now, with yesterday’s WSJ report, we witness the tawdry spectacle of large numbers of people who for years were fine with, responsible for, and even giddy about NSA mass surveillance suddenly objecting. Now they’ve learned that they themselves, or the officials of the foreign country they most love, have been caught up in this surveillance dragnet, and they can hardly contain their indignation. Overnight, privacy is of the highest value because now it’s their privacy, rather than just yours, that is invaded.

This reminds me of the 2013 story that the NSA eavesdropped on the cell phone of the German Chancellor Angela Merkel. Back then, I wrote:

Spying on foreign governments is what the NSA is supposed to do. Much more problematic, and dangerous, is that the NSA is spying on entire populations.

Greenwald said the same thing:

I’ve always argued that on the spectrum of spying stories, revelations about targeting foreign leaders is the least important, since that is the most justifiable type of espionage. Whether the U.S. should be surveilling the private conversations of officials of allied democracies is certainly worth debating, but, as I argued in my 2014 book, those “revelations … are less significant than the agency’s warrantless mass surveillance of whole populations” since “countries have spied on heads of state for centuries, including allies.”

And that’s the key point. I am less concerned about Angela Merkel than the other 82 million Germans that are being spied on, and I am less concerned about Benjamin Netanyahu than I am about the other 8 million people living in that country.

Over on Lawfare, Ben Wittes agrees:

There is absolutely nothing surprising about NSA’s activities here—or about the administration’s activities. There is no reason to expect illegality or impropriety. In fact, the remarkable aspect of this story is how constrained both the administration’s and the agency’s behavior appears to have been by rules and norms in exactly the fashion one would hope to see.

[…]

So let’s boil this down to brass tacks: NSA spied on a foreign leader at a time when his country had a major public foreign policy showdown with the President of the United States over a sharp differences between the two countries over Iran’s nuclearization—indeed, at a time when the US believed that leader was contemplating military action without advance notice to the United States. In the course of this surveillance, NSA incidentally collected communications involving members of Congress, who were being heavily lobbied by the Israeli government and Netanyahu personally. There is no indication that the members of Congress were targeted for collection. Moreover, there’s no indication that the rules that govern incidental collection involving members of Congress were not followed. The White House, for its part, appears to have taken a hands-off approach, directing NSA to follow its own policies about what to report, even on a sensitive matter involving delicate negotiations in a tense period with an ally.

The words that really matter are “incidental collection.” I have no doubt that the NSA followed its own rules in that regard. The discussion we need to have is about whether those rules are the correct ones. Section 702 incidental collection is a huge loophole that allows the NSA to collect information on millions of innocent Americans.

Greenwald again:

This claim of “incidental collection” has always been deceitful, designed to mask the fact that the NSA does indeed frequently spy on the conversations of American citizens without warrants of any kind. Indeed, as I detailed here, the 2008 FISA law enacted by Congress had as one of its principal, explicit purposes allowing the NSA to eavesdrop on Americans’ conversations without warrants of any kind. “The principal purpose of the 2008 law was to make it possible for the government to collect Americans’ international communications—and to collect those communications without reference to whether any party to those communications was doing anything illegal,” the ACLU’s Jameel Jaffer said. “And a lot of the government’s advocacy is meant to obscure this fact, but it’s a crucial one: The government doesn’t need to ‘target’ Americans in order to collect huge volumes of their communications.”

If you’re a member of Congress, there are special rules that the NSA has to follow if you’re incidentally spied on:

Special safeguards for lawmakers, dubbed the “Gates Rule,” were put in place starting in the 1990s. Robert Gates, who headed the Central Intelligence Agency from 1991 to 1993, and later went on to be President Barack Obama’s Defense Secretary, required intelligence agencies to notify the leaders of the congressional intelligence committees whenever a lawmaker’s identity was revealed to an executive branch official.

If you’re a regular American citizen, don’t expect any such notification. Your information can be collected, searched, and then saved for later searching, without a warrant. And if you’re a common German, Israeli, or any other countries’ citizen, you have even fewer rights.

In 2014, I argued that we need to separate the NSA’s espionage mission against target agents for a foreign power from any broad surveillance of Americans. I still believe that. But more urgently, we need to reform Section 702 when it comes up for reauthorization in 2017.

EDITED TO ADD: A good article on the topic. And Marcy Wheeler’s interesting take.

Tags: , , , , ,

Posted on January 5, 2016 at 6:36 AM • View Comments

← Earlier EntriesLater Entries →

Sidebar photo of Bruce Schneier by Joe MacInnis.