Entries Tagged "complexity"

Page 1 of 1

Excellent Analysis of the Boeing 737 Max Software Problems

This is the best analysis of the software causes of the Boeing 737 MAX disasters that I have read.

Technically this is safety and not security; there was no attacker. But the fields are closely related and there are a lot of lessons for IoT security — and the security of complex socio-technical systems in general — in here.

EDITED TO ADD (4/30): A rebuttal of sorts.

EDITED TO ADD (5/13): The comments to this blog post are of particularly high quality, and I recommend them to anyone interested in the topic.

Posted on April 22, 2019 at 8:45 AMView Comments

Accountability as a Security System

At a CATO surveillance event last month, Ben Wittes talked about inherent presidential powers of surveillance with this hypothetical: “What should Congress have to say about the rules when Barack Obama wants to know what Vladimir Putin is talking about?” His answer was basically that Congress should have no say: “I think most people, going back to my Vladimir Putin question, would say that is actually an area of inherent presidential authority.” Edward Snowden, a surprise remote participant at the event, said the opposite, although using the courts in general rather than specifically Congress as his example. “…there is no court in the world — well, at least, no court outside Russia — who would not go, ‘This man is an agent of the foreign government. I mean, he’s the head of the government.’ Of course, they will say, ‘this guy has access to some kind of foreign intelligence value. We’ll sign the warrant for him.'”

There’s a principle here worth discussing at length. I’m not talking about the legal principle, as in what kind of court should oversee US intelligence collection. I’m not even talking about the constitutional principle, as in what are the US president’s inherent powers. I am talking about the philosophical principle: what sorts of secret unaccountable actions do we want individuals to be able to take on behalf of their country?

Put that way, I think the answer is obvious: as little as possible.

I am not a lawyer or a political scientist. I am a security technologist. And to me, the separation of powers and the checks and balances written into the US constitution are a security system. The more Barack Obama can do by himself in secret, the more power he has — and the more dangerous that is to all of us. By limiting the actions individuals and groups can take on their own, and forcing differing institutions to approve the actions of each other, the system reduces the ability for those in power to abuse their power. It holds them accountable.

We have enshrined the principle of different groups overseeing each other in many of our social and political systems. The courts issue warrants, limiting police power. Independent audit companies verify corporate balance sheets, limiting corporate power. And the executive, the legislative, and the judicial branches of government get to have their say in our laws. Sometimes accountability takes the form of prior approval, and sometimes it takes the form of ex post facto review. It’s all inefficient, of course, but it’s an inefficiency we accept because it makes us all safer.

While this is a fine guiding principle, it quickly falls apart in the practicalities of running a modern government. It’s just not possible to run a country where every action is subject to review and approval. The complexity of society, and the speed with which some decisions have to be made, can require unilateral actions. So we make allowances. Congress passes broad laws, and agencies turn them into detailed rules and procedures. The president is the commander in chief of the entire US military when it comes time to fight wars. Policeman have a lot of discretion on their own on the beat. And we only get to vote elected officials in and out of office every two, four, or six years.

The thing is, we can do better today. I’ve often said that the modern constitutional democracy is the best form of government mid-18th-century technology could produce. Because both communications and travel were difficult and expensive, it made sense for geographically proximate groups of people to choose one representative to go all the way over there and act for them over a long block of time.

Neither of these two limitations is true today. Travel is both cheap and easy, and communications are so cheap and easy as to be virtually free. Video conferencing and telepresence allow people to communicate without traveling. Surely if we were to design a democratic government today, we would come up with better institutions than the ones we are stuck with because of history.

And we can come up with more granular systems of checks and balances. So, yes, I think we would have a better government if a court had to approve all surveillance actions by the president, including those against Vladimir Putin. And today it might be possible to have a court do just that. Wittes argues that making some of these changes is impossible, given the current US constitution. He may be right, but that doesn’t mean they’re not good ideas.

Of course, the devil is always in the details. Efficiency is still a powerful counterargument. The FBI has procedures for temporarily bypassing prior approval processes if speed is essential. And granularity can still be a problem. Every bullet fired by the US military can’t be subject to judicial approval or even a military court, even though every bullet fired by a US policeman is — at least in theory — subject to judicial review. And while every domestic surveillance decision made by the police and the NSA is (also in theory) subject to judicial approval, it’s hard to know whether this can work for international NSA surveillance decisions until we try.

We are all better off now that many of the NSA’s surveillance programs have been made public and are being debated in Congress and in the media — although I had hoped for more congressional action — and many of the FISA Court’s formerly secret decisions on surveillance are being made public. But we still have a long way to go, and it shouldn’t take someone like Snowden to force at least some openness to happen.

This essay previously appeared on Lawfare.com, where Ben Wittes responded.

Posted on January 20, 2015 at 6:24 AMView Comments

Complexity and Security

I have written about complexity and security for over a decade now (for example, this from 1999). Here’s the results of a survey that confirms this:

Results showed that more than half of the survey respondents from mid-sized (identified as 50-2500 employees) and enterprise organizations (identified as 2500+ employees) stated that complex policies ultimately led to a security breach, system outage or both.

Usual caveats for this sort of thing apply. The survey is only among 127 people — I can’t find data on what percentage replied. The numbers are skewed because only those that chose to reply were counted. And the results are based on self-reported replies: no way to verify them.

But still.

Posted on January 29, 2013 at 6:32 AMView Comments

Complexity and Terrorism Investigations

Good article on how complexity greatly limits the effectiveness of terror investigations. The stories of wasted resources are all from the UK, but the morals are universal.

The Committee’s report accepts that the increasing number of investigations, together with their increasing complexity, will make longer detention inevitable in the future. The core calculation is essentially the one put forward by the police and accepted by the Government – technology has been an enabler for international terrorism, with email, the Internet and mobile telephony producing wide, diffuse, international networks. The data on hard drives and mobile phones needs to be examined, contacts need to be investigated and their data examined, and in the case of an incident, vast amounts of CCTV records need to be gone through. As more and more of this needs to be done, the time taken to do it will obviously climb, and as it’s ‘necessary’ to detain the new breed of terrorist early in the investigation before he can strike, more time will be needed between arrest and charge in order to build a case.

All of which is, as far as it goes, logical. But take it a little further and the inherent futility of the route becomes apparent – ultimately, probably quite soon, the volume of data overwhelms the investigators and infinite time is needed to analyse all of it. And the less developed the plot is at the time the suspects are pulled in, the greater the number of possible outcomes (things they ‘might’ be planning) that will need to be chased-up. Short of the tech industry making the breakthrough into machine intelligence that will effectively do the analysis for them (which is a breakthrough the snake-oil salesmen suggest, and dopes in Government believe, has been achieved already), the approach itself is doomed. Essentially, as far as data is concerned police try to ‘collar the lot’ and then through analysis, attempt to build the most complete picture of a case that is possible. Use of initiative, experience and acting on probabilities will tend to be pressured out of such systems, and as the data volumes grow the result will tend to be teams of disempowered machine minders chained to a system that has ground to a halt. This effect is manifesting itself visibly across UK Government systems in general, we humbly submit. But how long will it take them to figure this out?

[…]

There is clearly a major problem for the security services in distinguishing disaffected talk from serious planning, and in deciding when an identified group constitutes a real threat. But the current technology-heavy approach to the threat doesn’t make a great deal of sense, because it produces very large numbers of suspects who are not and never will be a serious threat. Quantities of these suspects will nevertheless be found to be guilty of something, and along the way large amounts of investigative resource will have been expended to no useful purpose, aside from filling up 90 days. Overreaction to suggestions of CBRN threats is similarly counter-productive, because it makes it more likely that nascent groups will, just like the police, misunderstand the capabilities of the weapons, and start trying to research and build them. Mischaracterising the threat by inflating early, inexpert efforts as ‘major plots’ meanwhile fosters a climate of fear and ultimately undermines public confidence in the security services.

The oft-used construct, “the public would never forgive us if…” is a cop-out. It’s a spurious justification for taking the ‘collar the lot’ approach, throwing resources at it, ducking out of responsibility and failing to manage. Getting back to basics, taking ownership and telling the public the truth is more honest, and has some merit. A serious terror attack needs intent, attainable target and capability, the latter being the hard bit amateurs have trouble achieving without getting spotted along the way. Buying large bags of fertiliser if you’re not known to the vendor and you don’t look in the slightest bit like a farmer is going to put you onto MI5’s radar, and despite what it says on a lot of web sites, making your own explosives if you don’t know what you’re doing is a good way of blowing yourself up before you intended to. If disaffected youth had a more serious grasp of these realities, and had heard considerably more sense about the practicalities, then it’s quite possible that fewer of them would persist with their terror studies. Similarly, if the general public had better knowledge it would be better placed to spot signs of bomb factories. Bleached hair, dead plants, large numbers of peroxide containers? It could surely have been obvious.

Posted on July 14, 2006 at 7:25 AMView Comments

The Problem with Electronic Voting Machines

In the aftermath of the U.S.’s 2004 election, electronic voting machines are again in the news. Computerized machines lost votes, subtracted votes instead of adding them, and doubled votes. Because many of these machines have no paper audit trails, a large number of votes will never be counted. And while it is unlikely that deliberate voting-machine fraud changed the result of the presidential election, the Internet is buzzing with rumors and allegations of fraud in a number of different jurisdictions and races. It is still too early to tell if any of these problems affected any individual elections. Over the next several weeks we’ll see whether any of the information crystallizes into something significant.

The U.S has been here before. After 2000, voting machine problems made international headlines. The government appropriated money to fix the problems nationwide. Unfortunately, electronic voting machines — although presented as the solution — have largely made the problem worse. This doesn’t mean that these machines should be abandoned, but they need to be designed to increase both their accuracy, and peoples’ trust in their accuracy. This is difficult, but not impossible.

Before I can discuss electronic voting machines, I need to explain why voting is so difficult. Basically, a voting system has four required characteristics:

  1. Accuracy. The goal of any voting system is to establish the intent of each individual voter, and translate those intents into a final tally. To the extent that a voting system fails to do this, it is undesirable. This characteristic also includes security: It should be impossible to change someone else’s vote, ballot stuff, destroy votes, or otherwise affect the accuracy of the final tally.

  2. Anonymity. Secret ballots are fundamental to democracy, and voting systems must be designed to facilitate voter anonymity.

  3. Scalability. Voting systems need to be able to handle very large elections. One hundred million people vote for president in the United States. About 372 million people voted in India’s June elections, and over 115 million in Brazil’s October elections. The complexity of an election is another issue. Unlike many countries where the national election is a single vote for a person or a party, a United States voter is faced with dozens of individual election: national, local, and everything in between.

  4. Speed. Voting systems should produce results quickly. This is particularly important in the United States, where people expect to learn the results of the day’s election before bedtime. It’s less important in other countries, where people don’t mind waiting days — or even weeks — before the winner is announced.

Through the centuries, different technologies have done their best. Stones and pot shards dropped in Greek vases gave way to paper ballots dropped in sealed boxes. Mechanical voting booths, punch cards, and then optical scan machines replaced hand-counted ballots. New computerized voting machines promise even more efficiency, and Internet voting even more convenience.

But in the rush to improve speed and scalability, accuracy has been sacrificed. And to reiterate: accuracy is not how well the ballots are counted by, for example, a punch-card reader. It’s not how the tabulating machine deals with hanging chads, pregnant chads, or anything like that. Accuracy is how well the process translates voter intent into properly counted votes.

Technologies get in the way of accuracy by adding steps. Each additional step means more potential errors, simply because no technology is perfect. Consider an optical-scan voting system. The voter fills in ovals on a piece of paper, which is fed into an optical-scan reader. The reader senses the filled-in ovals and tabulates the votes. This system has several steps: voter to ballot to ovals to optical reader to vote tabulator to centralized total.

At each step, errors can occur. If the ballot is confusing, then some voters will fill in the wrong ovals. If a voter doesn’t fill them in properly, or if the reader is malfunctioning, then the sensor won’t sense the ovals properly. Mistakes in tabulation — either in the machine or when machine totals get aggregated into larger totals — also cause errors. A manual system — tallying the ballots by hand, and then doing it again to double-check — is more accurate simply because there are fewer steps.

The error rates in modern systems can be significant. Some voting technologies have a 5% error rate: one in twenty people who vote using the system don’t have their votes counted properly. This system works anyway because most of the time errors don’t matter. If you assume that the errors are uniformly distributed — in other words, that they affect each candidate with equal probability — then they won’t affect the final outcome except in very close races. So we’re willing to sacrifice accuracy to get a voting system that will more quickly handle large and complicated elections. In close races, errors can affect the outcome, and that’s the point of a recount. A recount is an alternate system of tabulating votes: one that is slower (because it’s manual), simpler (because it just focuses on one race), and therefore more accurate.

Note that this is only true if everyone votes using the same machines. If parts of town that tend to support candidate A use a voting system with a higher error rate than the voting system used in parts of town that tend to support candidate B, then the results will be skewed against candidate A. This is an important consideration in voting accuracy, although tangential to the topic of this essay.

With this background, the issue of computerized voting machines becomes clear. Actually, “computerized voting machines” is a bad choice of words. Many of today’s voting technologies involve computers. Computers tabulate both punch-card and optical-scan machines. The current debate centers around all-computer voting systems, primarily touch-screen systems, called Direct Record Electronic (DRE) machines. (The voting system used in India’s most recent election — a computer with a series of buttons — is subject to the same issues.) In these systems the voter is presented with a list of choices on a screen, perhaps multiple screens if there are multiple elections, and he indicates his choice by touching the screen. These machines are easy to use, produce final tallies immediately after the polls close, and can handle very complicated elections. They also can display instructions in different languages and allow for the blind or otherwise handicapped to vote without assistance.

They’re also more error-prone. The very same software that makes touch-screen voting systems so friendly also makes them inaccurate. And even worse, they’re inaccurate in precisely the worst possible way.

Bugs in software are commonplace, as any computer user knows. Computer programs regularly malfunction, sometimes in surprising and subtle ways. This is true for all software, including the software in computerized voting machines. For example:

In Fairfax County, VA, in 2003, a programming error in the electronic voting machines caused them to mysteriously subtract 100 votes from one particular candidates’ totals.

In San Bernardino County, CA in 2001, a programming error caused the computer to look for votes in the wrong portion of the ballot in 33 local elections, which meant that no votes registered on those ballots for that election. A recount was done by hand.

In Volusia County, FL in 2000, an electronic voting machine gave Al Gore a final vote count of negative 16,022 votes.

The 2003 election in Boone County, IA, had the electronic vote-counting equipment showing that more than 140,000 votes had been cast in the Nov. 4 municipal elections. The county has only 50,000 residents and less than half of them were eligible to vote in this election.

There are literally hundreds of similar stories.

What’s important about these problems is not that they resulted in a less accurate tally, but that the errors were not uniformly distributed; they affected one candidate more than the other. This means that you can’t assume that errors will cancel each other out and not affect the election; you have to assume that any error will skew the results significantly.

Another issue is that software can be hacked. That is, someone can deliberately introduce an error that modifies the result in favor of his preferred candidate. This has nothing to do with whether the voting machines are hooked up to the Internet on election day. The threat is that the computer code could be modified while it is being developed and tested, either by one of the programmers or a hacker who gains access to the voting machine company’s network. It’s much easier to surreptitiously modify a software system than a hardware system, and it’s much easier to make these modifications undetectable.

A third issue is that these problems can have further-reaching effects in software. A problem with a manual machine just affects that machine. A software problem, whether accidental or intentional, can affect many thousands of machines — and skew the results of an entire election.

Some have argued in favor of touch-screen voting systems, citing the millions of dollars that are handled every day by ATMs and other computerized financial systems. That argument ignores another vital characteristic of voting systems: anonymity. Computerized financial systems get most of their security from audit. If a problem is suspected, auditors can go back through the records of the system and figure out what happened. And if the problem turns out to be real, the transaction can be unwound and fixed. Because elections are anonymous, that kind of security just isn’t possible.

None of this means that we should abandon touch-screen voting; the benefits of DRE machines are too great to throw away. But it does mean that we need to recognize its limitations, and design systems that can be accurate despite them.

Computer security experts are unanimous on what to do. (Some voting experts disagree, but I think we’re all much better off listening to the computer security experts. The problems here are with the computer, not with the fact that the computer is being used in a voting application.) And they have two recommendations:

  1. DRE machines must have a voter-verifiable paper audit trails (sometimes called a voter-verified paper ballot). This is a paper ballot printed out by the voting machine, which the voter is allowed to look at and verify. He doesn’t take it home with him. Either he looks at it on the machine behind a glass screen, or he takes the paper and puts it into a ballot box. The point of this is twofold. One, it allows the voter to confirm that his vote was recorded in the manner he intended. And two, it provides the mechanism for a recount if there are problems with the machine.

  2. Software used on DRE machines must be open to public scrutiny. This also has two functions. One, it allows any interested party to examine the software and find bugs, which can then be corrected. This public analysis improves security. And two, it increases public confidence in the voting process. If the software is public, no one can insinuate that the voting system has unfairness built into the code. (Companies that make these machines regularly argue that they need to keep their software secret for security reasons. Don’t believe them. In this instance, secrecy has nothing to do with security.)

Computerized systems with these characteristics won’t be perfect — no piece of software is — but they’ll be much better than what we have now. We need to start treating voting software like we treat any other high-reliability system. The auditing that is conducted on slot machine software in the U.S. is significantly more meticulous than what is done to voting software. The development process for mission-critical airplane software makes voting software look like a slapdash affair. If we care about the integrity of our elections, this has to change.

Proponents of DREs often point to successful elections as “proof” that the systems work. That completely misses the point. The fear is that errors in the software — either accidental or deliberately introduced — can undetectably alter the final tallies. An election without any detected problems is no more a proof the system is reliable and secure than a night that no one broke into your house is proof that your door locks work. Maybe no one tried, or maybe someone tried and succeeded…and you don’t know it.

Even if we get the technology right, we still won’t be done. If the goal of a voting system is to accurately translate voter intent into a final tally, the voting machine is only one part of the overall system. In the 2004 U.S. election, problems with voter registration, untrained poll workers, ballot design, and procedures for handling problems resulted in far more votes not being counted than problems with the technology. But if we’re going to spend money on new voting technology, it makes sense to spend it on technology that makes the problem easier instead of harder.

This article originally appeared on openDemocracy.com.

Posted on November 10, 2004 at 9:15 AMView Comments

Getting Out the Vote: Why is it so hard to run an honest election?

Four years after the Florida debacle of 2000 and two years after Congress passed the Help America Vote Act, voting problems are again in the news: confusing ballots, malfunctioning voting machines, problems over who’s registered and who isn’t. All this brings up a basic question: Why is it so hard to run an election?

A fundamental requirement for a democratic election is a secret ballot, and that’s the first reason. Computers regularly handle multimillion-dollar financial transactions, but much of their security comes from the ability to audit the transactions after the fact and correct problems that arise. Much of what they do can be done the next day if the system is down. Neither of these solutions works for elections.

American elections are particularly difficult because they’re so complicated. One ballot might have 50 different things to vote on, all but one different in each state and many different in each district. It’s much easier to hold national elections in India, where everyone casts a single vote, than in the United States. Additionally, American election systems need to be able to handle 100 million voters in a single day — an immense undertaking in the best of circumstances.

Speed is another factor. Americans demand election results before they go to sleep; we won’t stand for waiting more than two weeks before knowing who won, as happened in India and Afghanistan this year.

To make matters worse, voting systems are used infrequently, at most a few times a year. Systems that are used every day improve because people familiarize themselves with them, discover mistakes and figure out improvements. It seems as if we all have to relearn how to vote every time we do it.

It should be no surprise that there are problems with voting. What’s surprising is that there aren’t more problems. So how to make the system work better?

— Simplicity: This is the key to making voting better. Registration should be as simple as possible. The voting process should be as simple as possible. Ballot designs should be simple, and they should be tested. The computer industry understands the science of user-interface — that knowledge should be applied to ballot design.

— Uniformity: Simplicity leads to uniformity. The United States doesn’t have one set of voting rules or one voting system. It has 51 different sets of voting rules — one for every state and the District of Columbia — and even more systems. The more systems are standardized around the country, the more we can learn from each other’s mistakes.

— Verifiability: Computerized voting machines might have a simple user interface, but complexity hides behind the screen and keyboard. To avoid even more problems, these machines should have a voter-verifiable paper ballot. This isn’t a receipt; it’s not something you take home with you. It’s a paper “ballot” with your votes — one that you verify for accuracy and then put in a ballot box. The machine provides quick tallies, but the paper is the basis for any recounts.

— Transparency: All computer code used in voting machines should be public. This allows interested parties to examine the code and point out errors, resulting in continually improving security. Any voting-machine company that claims its code must remain secret for security reasons is lying. Security in computer systems comes from transparency — open systems that pass public scrutiny — and not secrecy.

But those are all solutions for the future. If you’re a voter this year, your options are fewer. My advice is to vote carefully. Read the instructions carefully, and ask questions if you are confused. Follow the instructions carefully, checking every step as you go. Remember that it might be impossible to correct a problem once you’ve finished voting. In many states — including California — you can request a paper ballot if you have any worries about the voting machine.

And be sure to vote. This year, thousands of people are watching and waiting at the polls to help voters make sure their vote counts.

This essay originally appeared in the San Francisco Chronicle.

Also read Avi Rubin’s op-ed on the subject.

Posted on October 31, 2004 at 9:13 AMView Comments

Sidebar photo of Bruce Schneier by Joe MacInnis.