Page 1 of 2
Examples of dumb password rules.
There are some pretty bad disasters out there.
My worst experiences are with sites that have artificial complexity requirements that cause my personal password-generation systems to fail. Some of the systems on the list are even worse: when they fail they don’t tell you why, so you just have to guess until you get it right.
We know that complexity is the worst enemy of security, because it makes attack easier and defense harder. This becomes catastrophic as the effects of that attack become greater.
In A Hacker’s Mind (coming in February 2023), I write:
Our societal systems, in general, may have grown fairer and more just over the centuries, but progress isn’t linear or equitable. The trajectory may appear to be upwards when viewed in hindsight, but from a more granular point of view there are a lot of ups and downs. It’s a “noisy” process.
Technology changes the amplitude of the noise. Those near-term ups and downs are getting more severe. And while that might not affect the long-term trajectories, they drastically affect all of us living in the short term. This is how the twentieth century could—statistically—both be the most peaceful in human history and also contain the most deadly wars.
Ignoring this noise was only possible when the damage wasn’t potentially fatal on a global scale; that is, if a world war didn’t have the potential to kill everybody or destroy society, or occur in places and to people that the West wasn’t especially worried about. We can’t be sure of that anymore. The risks we face today are existential in a way they never have been before. The magnifying effects of technology enable short-term damage to cause long-term planet-wide systemic damage. We’ve lived for half a century under the potential specter of nuclear war and the life-ending catastrophe that could have been. Fast global travel allowed local outbreaks to quickly become the COVID-19 pandemic, costing millions of lives and billions of dollars while increasing political and social instability. Our rapid, technologically enabled changes to the atmosphere, compounded through feedback loops and tipping points, may make Earth much less hospitable for the coming centuries. Today, individual hacking decisions can have planet-wide effects. Sociobiologist Edward O. Wilson once described the fundamental problem with humanity is that “we have Paleolithic emotions, medieval institutions, and godlike technology.”
Technology could easily get to the point where the effects of a successful attack could be existential. Think biotech, nanotech, global climate change, maybe someday cyberattack—everything that people like Nick Bostrom study. In these areas, like everywhere else in past and present society, the technologies of attack develop faster the technologies of defending against attack. But suddenly, our inability to be proactive becomes fatal. As the noise due to technological power increases, we reach a threshold where a small group of people can irrecoverably destroy the species. The six-sigma guy can ruin it for everyone. And if they can, sooner or later they will. It’s possible that I have just explained the Fermi paradox.
This is from a court deposition:
Facebook’s stonewalling has been revealing on its own, providing variations on the same theme: It has amassed so much data on so many billions of people and organized it so confusingly that full transparency is impossible on a technical level. In the March 2022 hearing, Zarashaw and Steven Elia, a software engineering manager, described Facebook as a data-processing apparatus so complex that it defies understanding from within. The hearing amounted to two high-ranking engineers at one of the most powerful and resource-flush engineering outfits in history describing their product as an unknowable machine.
The special master at times seemed in disbelief, as when he questioned the engineers over whether any documentation existed for a particular Facebook subsystem. “Someone must have a diagram that says this is where this data is stored,” he said, according to the transcript. Zarashaw responded: “We have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” He quickly added, “For what it’s worth, this is terrifying to me when I first joined as well.”
[…]
Facebook’s inability to comprehend its own functioning took the hearing up to the edge of the metaphysical. At one point, the court-appointed special master noted that the “Download Your Information” file provided to the suit’s plaintiffs must not have included everything the company had stored on those individuals because it appears to have no idea what it truly stores on anyone. Can it be that Facebook’s designated tool for comprehensively downloading your information might not actually download all your information? This, again, is outside the boundaries of knowledge.
“The solution to this is unfortunately exactly the work that was done to create the DYI file itself,” noted Zarashaw. “And the thing I struggle with here is in order to find gaps in what may not be in DYI file, you would by definition need to do even more work than was done to generate the DYI files in the first place.”
The systemic fogginess of Facebook’s data storage made answering even the most basic question futile. At another point, the special master asked how one could find out which systems actually contain user data that was created through machine inference.
“I don’t know,” answered Zarashaw. “It’s a rather difficult conundrum.”
I’m not surprised. These systems are so complex that no humans understand them anymore. That allows us to do things we couldn’t do otherwise, but it’s also a problem.
EDITED TO ADD: Another article.
I’ve been saying that complexity is the worst enemy of security for a long time now. (Here’s me in 1999.) And it’s been true for a long time.
In 2018, Thomas Dullien of Google’s Project Zero talked about “cheap complexity.” Andrew Appel summarizes:
The anomaly of cheap complexity. For most of human history, a more complex device was more expensive to build than a simpler device. This is not the case in modern computing. It is often more cost-effective to take a very complicated device, and make it simulate simplicity, than to make a simpler device. This is because of economies of scale: complex general-purpose CPUs are cheap. On the other hand, custom-designed, simpler, application-specific devices, which could in principle be much more secure, are very expensive.
This is driven by two fundamental principles in computing: Universal computation, meaning that any computer can simulate any other; and Moore’s law, predicting that each year the number of transistors on a chip will grow exponentially. ARM Cortex-M0 CPUs cost pennies, though they are more powerful than some supercomputers of the 20th century.
The same is true in the software layers. A (huge and complex) general-purpose operating system is free, but a simpler, custom-designed, perhaps more secure OS would be very expensive to build. Or as Dullien asks, “How did this research code someone wrote in two weeks 20 years ago end up in a billion devices?”
This is correct. Today, it’s easier to build complex systems than it is to build simple ones. As recently as twenty years ago, if you wanted to build a refrigerator you would create custom refrigerator controller hardware and embedded software. Today, you just grab some standard microcontroller off the shelf and write a software application for it. And that microcontroller already comes with an IP stack, a microphone, a video port, Bluetooth, and a whole lot more. And since those features are there, engineers use them.
This is the best analysis of the software causes of the Boeing 737 MAX disasters that I have read.
Technically this is safety and not security; there was no attacker. But the fields are closely related and there are a lot of lessons for IoT security—and the security of complex socio-technical systems in general—in here.
EDITED TO ADD (4/30): A rebuttal of sorts.
EDITED TO ADD (5/13): The comments to this blog post are of particularly high quality, and I recommend them to anyone interested in the topic.
For many years, I have said that complexity is the worst enemy of security. At CyCon earlier this month, Thomas Dullien gave an excellent talk on the subject with far more detail than I’ve ever provided. Video. Slides.
At a CATO surveillance event last month, Ben Wittes talked about inherent presidential powers of surveillance with this hypothetical: “What should Congress have to say about the rules when Barack Obama wants to know what Vladimir Putin is talking about?” His answer was basically that Congress should have no say: “I think most people, going back to my Vladimir Putin question, would say that is actually an area of inherent presidential authority.” Edward Snowden, a surprise remote participant at the event, said the opposite, although using the courts in general rather than specifically Congress as his example. “…there is no court in the world—well, at least, no court outside Russia—who would not go, ‘This man is an agent of the foreign government. I mean, he’s the head of the government.’ Of course, they will say, ‘this guy has access to some kind of foreign intelligence value. We’ll sign the warrant for him.'”
There’s a principle here worth discussing at length. I’m not talking about the legal principle, as in what kind of court should oversee US intelligence collection. I’m not even talking about the constitutional principle, as in what are the US president’s inherent powers. I am talking about the philosophical principle: what sorts of secret unaccountable actions do we want individuals to be able to take on behalf of their country?
Put that way, I think the answer is obvious: as little as possible.
I am not a lawyer or a political scientist. I am a security technologist. And to me, the separation of powers and the checks and balances written into the US constitution are a security system. The more Barack Obama can do by himself in secret, the more power he has—and the more dangerous that is to all of us. By limiting the actions individuals and groups can take on their own, and forcing differing institutions to approve the actions of each other, the system reduces the ability for those in power to abuse their power. It holds them accountable.
We have enshrined the principle of different groups overseeing each other in many of our social and political systems. The courts issue warrants, limiting police power. Independent audit companies verify corporate balance sheets, limiting corporate power. And the executive, the legislative, and the judicial branches of government get to have their say in our laws. Sometimes accountability takes the form of prior approval, and sometimes it takes the form of ex post facto review. It’s all inefficient, of course, but it’s an inefficiency we accept because it makes us all safer.
While this is a fine guiding principle, it quickly falls apart in the practicalities of running a modern government. It’s just not possible to run a country where every action is subject to review and approval. The complexity of society, and the speed with which some decisions have to be made, can require unilateral actions. So we make allowances. Congress passes broad laws, and agencies turn them into detailed rules and procedures. The president is the commander in chief of the entire US military when it comes time to fight wars. Policeman have a lot of discretion on their own on the beat. And we only get to vote elected officials in and out of office every two, four, or six years.
The thing is, we can do better today. I’ve often said that the modern constitutional democracy is the best form of government mid-18th-century technology could produce. Because both communications and travel were difficult and expensive, it made sense for geographically proximate groups of people to choose one representative to go all the way over there and act for them over a long block of time.
Neither of these two limitations is true today. Travel is both cheap and easy, and communications are so cheap and easy as to be virtually free. Video conferencing and telepresence allow people to communicate without traveling. Surely if we were to design a democratic government today, we would come up with better institutions than the ones we are stuck with because of history.
And we can come up with more granular systems of checks and balances. So, yes, I think we would have a better government if a court had to approve all surveillance actions by the president, including those against Vladimir Putin. And today it might be possible to have a court do just that. Wittes argues that making some of these changes is impossible, given the current US constitution. He may be right, but that doesn’t mean they’re not good ideas.
Of course, the devil is always in the details. Efficiency is still a powerful counterargument. The FBI has procedures for temporarily bypassing prior approval processes if speed is essential. And granularity can still be a problem. Every bullet fired by the US military can’t be subject to judicial approval or even a military court, even though every bullet fired by a US policeman is—at least in theory—subject to judicial review. And while every domestic surveillance decision made by the police and the NSA is (also in theory) subject to judicial approval, it’s hard to know whether this can work for international NSA surveillance decisions until we try.
We are all better off now that many of the NSA’s surveillance programs have been made public and are being debated in Congress and in the media—although I had hoped for more congressional action—and many of the FISA Court’s formerly secret decisions on surveillance are being made public. But we still have a long way to go, and it shouldn’t take someone like Snowden to force at least some openness to happen.
This essay previously appeared on Lawfare.com, where Ben Wittes responded.
Good summary list. It’s not directly about security, but it’s all fundamentally about security. Any real-world security system is inherently complex. I wrote about this long ago in Beyond Fear.
I have written about complexity and security for over a decade now (for example, this from 1999). Here’s the results of a survey that confirms this:
Results showed that more than half of the survey respondents from mid-sized (identified as 50-2500 employees) and enterprise organizations (identified as 2500+ employees) stated that complex policies ultimately led to a security breach, system outage or both.
Usual caveats for this sort of thing apply. The survey is only among 127 people—I can’t find data on what percentage replied. The numbers are skewed because only those that chose to reply were counted. And the results are based on self-reported replies: no way to verify them.
But still.
Sidebar photo of Bruce Schneier by Joe MacInnis.