Hacking Airplanes

Imagine this: A terrorist hacks into a commercial airplane from the ground, takes over the controls from the pilots and flies the plane into the ground. It sounds like the plot of some “Die Hard” reboot, but it’s actually one of the possible scenarios outlined in a new Government Accountability Office report on security vulnerabilities in modern airplanes.

It’s certainly possible, but in the scheme of Internet risks I worry about, it’s not very high. I’m more worried about the more pedestrian attacks against more common Internet-connected devices. I’m more worried, for example, about a multination cyber arms race that stockpiles capabilities such as this, and prioritizes attack over defense in an effort to gain relative advantage. I worry about the democratization of cyberattack techniques, and who might have the capabilities currently reserved for nation-states. And I worry about a future a decade from now if these problems aren’t addressed.

First, the airplanes. The problem the GAO identifies is one computer security experts have talked about for years. Newer planes such as the Boeing 787 Dreamliner and the Airbus A350 and A380 have a single network that is used both by pilots to fly the plane and passengers for their Wi-Fi connections. The risk is that a hacker sitting in the back of the plane, or even one on the ground, could use the Wi-Fi connection to hack into the avionics and then remotely fly the plane.

The report doesn’t explain how someone could do this, and there are currently no known vulnerabilities that a hacker could exploit. But all systems are vulnerable—we simply don’t have the engineering expertise to design and build perfectly secure computers and networks—so of course we believe this kind of attack is theoretically possible.

Previous planes had separate networks, which is much more secure.

As terrifying as this movie-plot threat is—and it has been the plot of several recent works of fiction—this is just one example of an increasingly critical problem: As the computers already critical to running our infrastructure become connected, our vulnerability to cyberattack grows. We’ve already seen vulnerabilities in baby monitors, cars, medical equipment and all sorts of other Internet-connected devices. In February, Toyota recalled 1.9 million Prius cars because of a software vulnerability. Expect similar vulnerabilities in our smart thermostats, smart light bulbs and everything else connected to the smart power grid. The Internet of Things will bring computers into every aspect of our life and society. Those computers will be on the network and will be vulnerable to attack.

And because they’ll all be networked together, a vulnerability in one device will affect the security of everything else. Right now, a vulnerability in your home router can compromise the security of your entire home network. A vulnerability in your Internet-enabled refrigerator can reportedly be used as a launching pad for further attacks.

Future attacks will be exactly like what’s happening on the Internet today with your computer and smartphones, only they will be with everything. It’s all one network, and it’s all critical infrastructure.

Some of these attacks will require sufficient budget and organization to limit them to nation-state aggressors. But that’s hardly comforting. North Korea is last year believed to have launched a massive cyberattack against Sony Pictures. Last month, China used a cyberweapon called the “Great Cannon” against the website GitHub. In 2010, the U.S. and Israeli governments launched a sophisticated cyberweapon called Stuxnet against the Iranian Natanz nuclear power plant; it used a series of vulnerabilities to cripple centrifuges critical for separating nuclear material. In fact, the United States has done more to weaponize the Internet than any other country.

Governments only have a fleeting advantage over everyone else, though. Today’s top-secret National Security Agency programs become tomorrow’s Ph.D. theses and the next day’s hacker’s tools. So while remotely hacking the 787 Dreamliner’s avionics might be well beyond the capabilities of anyone except Boeing engineers today, that’s not going to be true forever.

What this all means is that we have to start thinking about the security of the Internet of Things—whether the issue in question is today’s airplanes or tomorrow’s smart clothing. We can’t repeat the mistakes of the early days of the PC and then the Internet, where we initially ignored security and then spent years playing catch-up. We have to build security into everything that is going to be connected to the Internet.

This is going to require both significant research and major commitments by companies. It’s also going to require legislation mandating certain levels of security on devices connecting to the Internet, and at network providers that make the Internet work. This isn’t something the market can solve on its own, because there are just too many incentives to ignore security and hope that someone else will solve it.

As a nation, we need to prioritize defense over offense. Right now, the NSA and U.S. Cyber Command have a strong interest in keeping the Internet insecure so they can better eavesdrop on and attack our enemies. But this prioritization cuts both ways: We can’t leave others’ networks vulnerable without also leaving our own vulnerable. And as one of the most networked countries on the planet, we are highly vulnerable to attack. It would be better to focus the NSA’s mission on defense and harden our infrastructure against attack.

Remember the GAO’s nightmare scenario: A hacker on the ground exploits a vulnerability in the airplane’s Wi-Fi system to gain access to the airplane’s network. Then he exploits a vulnerability in the firewall that separates the passengers’ network from the avionics to gain access to the flight controls. Then he uses other vulnerabilities both to lock the pilots out of the cockpit controls and take control of the plane himself.

It’s a scenario made possible by insecure computers and insecure networks. And while it might take a government-led secret project on the order of Stuxnet to pull it off today, that won’t always be true.

Of course, this particular movie-plot threat might never become a real one. But it is almost certain that some equally unlikely scenario will. I just hope we have enough security expertise to deal with whatever it ends up being.

This essay originally appeared on CNN.com.

EDITED TO ADD: News articles.

Tags: , , , , ,

Posted on April 21, 2015 at 1:40 PM • 80 Comments

Comments

Nicolas George • April 21, 2015 2:17 PM

“it has been the plot of several recent works of fiction”

It was not the core of the plot itself, but the reason the Battlestar Galactica escaped is just that: it was the only non-networked ship.

“require legislation mandating certain levels of security on devices connecting to the Internet”

I do not think the legislation needs to mandate the level of security. This is too abstract and impossible to measure. The core of the problem is that software is still under the regime of so-called “intellectual property” (there are a few exceptions for software, but only for the commercial use, the copyright, when an employee codes for a company, the copyright usually goes to the company).

The basic principle is still that software is a “creation of the mind”, like music, novels, paintings. A painting is not required by law to work; a novel is not required to be secure. Change that, and you have the incentive to make the Internet of things secure.

If a vendor of home appliances is liable for the cost of damages caused by an exploit, it will invest in security audits. If the CEO of a company that sells pacemakers can go to prison as an accessory to murder because a vulnerability was used, they will take security seriously.

J Milner • April 21, 2015 2:27 PM

Does anyone have more information on why the planes don’t have physically separate networks? I know it’s probably related to cost, but I’m looking for a better reason. It seems like it would be obvious design to use separate networks. It’s a nightmare (think about QOS issues when someone’s downloading a video and you’re trying to control the elevators). I don’t understand why they would do it that way. Are they both sharing the network uplink maybe? I could get that they have mostly separate networks with a single peering point for the uplink, but not a single network.

J Milner • April 21, 2015 2:31 PM

Update to my comment, from a Wired article:

“Boeing spokeswoman Lori Gunter told WIRED in 2008 that the company did indeed design a solution to address the FAA concerns. She wouldn’t go into detail about how Boeing was tackling the problem but said Boeing was employing a combination of solutions that involved some physical air-gapping of the networks as well as software firewalls. “There are places where the networks are not touching, and there are places where they are,” she had said.”

So it sounds like they do have separate networks, with peering points. That makes more sense (still scary re: security, but not as crazy on the engineering side)

http://www.wired.com/2015/04/hackers-commandeer-new-planes-passenger-wi-fi/

rgaff • April 21, 2015 2:46 PM

@J Milner

Perhaps you’ve never worked at a company where a PR or Marketing person slapped a few “techie” words together in a non-specific way and technically lied about what they were doing? I have. I told them they were lying to the customers, and explained how, and I was almost fired for it. Instead, the entire engineering dept quit at once 2 months later.

Let’s just say what you just quoted sounds very familiar. (No, I did not work at that company, I just have experience with that kind of rhetoric)

dvv • April 21, 2015 3:44 PM

“Newer planes such as the Boeing 787 Dreamliner and the Airbus A350 and A380 have a single network that is used both by pilots to fly the plane and passengers for their Wi-Fi connections.”

Seriously?

Andrew Wallace • April 21, 2015 3:47 PM

As I have already mentioned the RQ-170 incident. The vulnerability of military manned and unmanned aircraft is far more developed than commercial airlines.

We have seen instances of military cyber hi jacks but nothing of civil aviation. The reason is simple, there is not a desire to hi jack a commercial liner in comparison to military assets.

The capability is out there among cyber ranks to interfere with civil avionics, though the bad guys e.g Islamic terrorists don’t have their hands on that stuff yet and we intend to keep it that way.

Happy flying chaps.

Another Kevin • April 21, 2015 4:20 PM

I’d actually be afraid that there is a path to seize control of the aircraft from the ground, designed in but not yet disclosed, with the idea being to wrest control from the flight crew in the event of a hijacking. Of course, the most likely outcome of that is that someone, either a terrorist, a turned insider, or just an insider gone insane, will use that feature to wrest control of a plane for nefarious purposes.

That seems to be a major avenue of attack nowadays: build in “security” features whose purpose is to keep users from being in control of their own devices, and then act surprised when evildoers subvert the features to wrest control of the devices away from their owners.

Chris • April 21, 2015 4:33 PM

So, “we simply don’t have the engineering expertise to design and build perfectly secure computers and networks,” and yet, “We have to build security into everything that is going to be connected to the Internet.” There’s a contradiction here.

Later in your essay, it sounds like you’re proposing a solution to the contradiction: Congress should change the NSA’s mission; at which time the NSA will wave their magic wand and make our software secure. I don’t believe their capabilities are so far ahead of ours that this is possible.

Bruce, can you be more specific about how we should go about making our software secure? It’s comparatively easy to find vulnerabilities in crypto, software, processes, or people. It’s more difficult to give useful advice on building invulnerable software.

Has anybody ever made a secure, non-trivial software product?

Andrew Wallace • April 21, 2015 4:39 PM

“Another Kevin • April 21, 2015 4:20 PM

I’d actually be afraid that there is a path to seize control of the aircraft from the ground, designed in but not yet disclosed.”

I would go with the theory that the new generation of planes have diagnostic tool capability to report faults to the engineering team.

Andrew

Owen • April 21, 2015 5:00 PM

@another kevin

I’ve thought about a “ground control override” and decided it is probably best to give the most powerful override to someone who must die himself in order to crash the plane.

Ken • April 21, 2015 5:23 PM

One of the challenges of the security research community, especially when they cross over from classic IT security into the Cyber-physical Systems realm is a lack of detailed understanding of the domain. This leads to some rather extravagant claims that are grabbed by the media and made even more wildly speculative. I have seen it in the medical device world, and this seems to be a parallel case in avionics systems design.

Two caveats – one, I never say never, especially when it comes to the potential of vulnerabilities in complex systems. Two, I have worked in detail in the avionics industry, but not specifically on the A380 or B787, and not in the space of the bridge between the In-Flight Entertainment (IFE) and Satcom systems.

That said, I have done some work with the network technology on these planes. While they use Ethernet physical layers and standard Enet frame formats, everything else is quite customized. Go search for “AFDX” on Google – stands for Aircraft Full Duplex Ethernet – standardized as ARINC 664. The avionics data networks on those two planes are built on this technology. One of its main characteristics is that all message flows are pre-declared, and their bandwidths pre-allocated. The source/destination of all message flows (called Virtual Links in AFDX-parlance) are statically stored in the switches, which validate the source/destination/port # of every frame that flows through them.

With this architecture you cannot get an arbitrary frame onto the network. If you try to inject one, the switch you are connected to will reject it unless it is a predeclared message. I highly doubt that the switches have been programmed to accept write messages sourced from the IFE system into an avionics computer.

The other way is likely – give that IFE equipment (and possibly Cabin systems for the flight attendants) may need information like aircraft position, speed, etc – at least what is needed to show the moving map to the passengers. If this researcher has claimed to have seen such messages from the WiFi network, he is foolish to think that if he changed them on that network that the aircraft would be affected one iota.

The primary needed connection is to/from the Satcom links. It is expensive and heavy to duplicate satcom radios and antennas, so I understand that the link is shared between IFE (e.g. passenger WiFi) and cockpit systems. But messages are not pre-programmed into the switches to accept control commands from a Satcom link and flow them to an autopilot, or engine control system and have them accept it as a legitimate input. I am highly skeptical of claims of being able to remotely control (crash) an airplane from the ground. I don’t believe the static connections are present. The worst case scenario I have seen amongst my aviation colleagues is a corrupted flight plan sent to the flight deck, accepted without review by the pilots, and then having the pilots not notice they aren’t going where they thought (with lots of alarms from the EGPWS if they are heading into terrain).

The nuances of how these on-board networks work are essential to understanding what risks are really there. From the open press reports on this “researchers” claims, I don’t get any sense that he understands how these aircraft have been designed. I would be interested the the backgrounds of all who were interviewed for the GAO report as well.

rgaff • April 21, 2015 6:52 PM

@Ken

Your detailed and specific technical description of the separations does much more to allay fears than stupid clueless PR people hand waving with vagueness…

Still, Bruce is right that ANY connection between avionics networks and the “rest of the world” (i.e. the internet, etc) is a potential attack point. Easiest way to not have those vulnerabilities is to not have those connections. At all.

And… should you (@Ken) now be hauled off by the FBI? You just exposed how airplane systems work! Maybe anyone who performs the searches you suggest on Google should? There seem to be a lot of stupid people (some in important positions in government) who think the answer is “yes” 🙁

AC #1 • April 21, 2015 6:58 PM

The so-called “Internet of Things” is scary. We may soon be completely surrounded by things that are insecure, networked, and don’t really need to be networked. The “IoT” should really be called the IoUT (Internet of Unsafe Things)!

I think we should start an awareness campaign. We can start by calling it what it is–IoUT–the Internet of Unsafe Things! Maybe some good will come of it when more people are aware of the potential danger.

Andrew Wallace • April 21, 2015 6:59 PM

Diagnostic data is invaluable. The diagnostic data flow is a feature design of the next generation of craft to help engineering teams prepare for a plane maintenance.

If I was to want to hack a plane the first thing I would do is download the Diagnostic data to check out the health of the plane and work from that.

Andrew

rgaff • April 21, 2015 7:06 PM

QUOTING @Andrew:

If I was to want to hack a plane the first thing I would do is download the Diagnostic data to check out the health of the plane and work from that.

OMG Andrew!!! SHHHHHHH You just told BAAAD people how to do it!!! prison for you!

Right? Isn’t that what you your very self keep arguing??? Sheesh man!

rgaff • April 21, 2015 7:40 PM

So if we don’t hear from Andrew for 4 hours should we assume he’s under the bright lights? 😛

dvv • April 21, 2015 7:53 PM

@rgaff IFE won’t be able to use the position, altitude, and speed data from the airplane’s avionics system then. And forget about any IFE links with any of the outside networks. In fact, forget about IFE altogether, as it’s known to bring airplanes down simply by being electric.

rgaff • April 21, 2015 8:06 PM

@dvv Really? “In Flight Entertainment” has been known to bring airplanes down??

I suspect you might be referring to FAA’s penchant for banning all electronics, for fear, as they bogusly claim to passengers, that switching anything on anywhere might suddenly make planes fall out of the sky. Making bogus claims and stupid rules based on them (or making stupid rules and then making up bogus claims to support them) is quite different from something being known to happen.

rgaff • April 21, 2015 8:10 PM

@dvv but you are absolutely correct, totally separating the flight systems from entertainment systems does mean passengers can’t be entertained by the flight data anywhere they want… aww poo. Let’s see… security vs entertainment… hmm.. tough choice.

rgaff • April 21, 2015 8:29 PM

To put my last two comments together could seem contradictory, let me explain: Commercial airplane avionics aren’t known to be hackable through the entertainment system… But generally connecting two networks in any way at all, is generally known to often have unforeseen interactions (read: “bugs” which are often “vulnerabilities” which is what hackers use), the moreso the more complex they are, and the more complex the interaction is. So the “knowns” is a little bit removed from planes directly falling out of the sky… so far anyway.

Nick P • April 21, 2015 8:48 PM

@ Ken

Nice post. The protocols they use are pretty interesting for security engineering reasons as well as their use cases.

@ All

The risks are not new or even theoretical. The industry and government already know about the issues as companies like Boeing were tasked with coming up with solutions. Further, the aircraft designers are using many tools to improve the safety and security of their critical software. DARPA’s HACMS program resulted in new technologies that were applied to a UAV. Air Force is doing another round of grants on improving security of military avionics.

So, that there’s a problem is known and there’s certainly work on it. My problem is that there aren’t enough experienced security engineers tasked to each part of the problem. Those currently involved are beholden to the industry mostly. There’s also not as much application of what we know as there could be. For instance, high assurance tech already exists to allow trusted and malicious traffic on the same wire with low risk of attack. It’s simple enough to be put in one inexpensive SOC. Yet, there’s still all this risk in most of our devices and infrastructure on something so basic.

The problem isn’t the technology. It’s incentives and politics.

65535 • April 21, 2015 11:14 PM

@ Ken

Your explanation helps a reduce fears. But, after looking at the specification it brings up even more questions [I will not delve into them because of the possibility of being put on some government list].

AFDX
https://en.wikipedia.org/wiki/AFDX

@ rgaff

“…should you (@Ken) now be hauled off by the FBI? You just exposed how airplane systems work!”

Your point is well noted. I don’t need any trouble from some TLA or an position on some government list.

“Bruce is right that ANY connection between avionics networks and the “rest of the world” (i.e. the internet, etc) is a potential attack point.”

I agree. It is a potential attack vector – Remote at the moment – but a potential attack vector.

@ Nick P

“@ Ken: Nice post. The protocols they use are pretty interesting for security engineering reasons as well as their use cases.”

It is interesting stuff.

“@ All: The risks are not new or even theoretical. The industry and government already know about the issues as companies like Boeing were tasked with coming up with solutions.”

I agree.

But, on the flip side of coin I am sure Vup@n, H@cking team and various state actors are working on this also – which in not a pleasant thought.

Ken • April 21, 2015 11:16 PM

@ Paul Marten – that patent is out of Boeing’s Huntington Beach Office, which is a former McD military/space facility I believe. It is highly unlikely, if that patent made it into product, that it is on Boeing’s commercial aircraft.

@ Nick P – are you working on one of the HACMS teams? I have some former colleagues who are working that project. I am about to start on one of the related projects out of the same contracting office.

Interesting links. I cut my teeth in avionics working on the B777 and a variety of biz jets and commuter jets. I agree with your observation that there aren’t enough people who are both competent in safety critical and security critical technologies. The capabilities are being built, but not always being deployed.

Ken • April 21, 2015 11:34 PM

@2^16-1 – If there are TLA lists for talking about technology, I would expect I have been on them for a long time. The details of what is used on these aircraft are widely known to those who spend time to look. There are textbooks and research papers and standards. Not exactly a hidden set of data.

I started searching for some of the references for the first roll out of AFDX on the A380 and found the following presentation where I was involved with all of the things they report on pages 9-12.

This stuff is all out there for the reading. I suspect that nation states are far more diligent searching out these sources than the “white hat” researcher. Security researchers should spend some time as well understanding what they are trying to hack. If I saw any technical details that showed some basic understanding of the constraints that safety critical systems are built to meet in their analysis, I would find their conclusions of more interest.

01 • April 22, 2015 1:18 AM

Can anyone of the smart, capable people here explain why, for the love of everything good, was the plane in question designed with a singular network and without hardware isolation?

Does having two god-damn physical networks that do not share any peering points at all increase cost that much?

Is there a pressing need for the pilot cockpit to have internet connectivity during flight ?

atken • April 22, 2015 1:43 AM

Ken
With ethernet based on ieee documents, there is a hardware flaw, after the first voltage rise, there is a set time delay set in the specs, if a nS before the pulse is sent and hald, you can by pass the first buffer, and get it directed to the irq line, just looking up hardware switch logic and ethernet cards map layout, with the ieee doc, you might see what I mean, I havnt looked at this bug for twelve yearsso its a bit hassy.

Wael • April 22, 2015 2:03 AM

The report doesn’t explain how someone could do this, and there are currently no known vulnerabilities that a hacker could exploit […] so of course we believe this kind of attack is theoretically possible.

Insightful statement with a lesson to be learned! Sometimes when I highlight a “Security weakness”, I get questioned: How would a hacker (or an “adversary”) break this? I usually say, I am not a hacker! I follow Security Principles! A violation of a principle is a sufficient weakness that needs to be rectified (mitigated.)

Same thing here: The report doesn’t have to explain how, but it’s likely someone will find an exploit to this weakness. It’s one of the reasons I say a “Security Architect” shouldn’t wear an attacker’s hat during the design of the system — this is a Pen tester’s job! Bottom line: Separate the networks! Because the controls on the passenger game console are so lousy for playing a game, let alone flying the hacked plane 🙂

rgaff • April 22, 2015 2:34 AM

@dvv ok I see where you’re going with that then.. yes, anything “electronic” on any aircraft could, by some kind of physical failure or malconstruction, at some point arc and cause fire, thus causing an airplane to crash. But every physical part of every physical thing in this universe will fail someday anyway, so if you don’t want to have any risk of dying from any of it, you’d better simply never be alive in the first place. This is different than a hacking risk however, the original subject of this post.

@65535 you just googled and looked up the specification? you’re definitely already on “the list” then… might as well just ask away your questions now… you already commented here too, so you were probably on “the list” beforehand anyway…

@all FYI my constant remonstrations about how we’re all on the lists, and how much people here should be hauled off according to the reasoning of the shills (including some of the shills themselves, go figure!), is not meant to make us all shut up and leave…. it’s meant to illustrate just how stupid the shills are and those who hired them.

@01 apparently, from what I’m reading into what @Ken says, it’s not that everything is on one shared network, it’s that one’s penchant for luxuries like “passengers seeing where their plane is on a map as they’re flying” trump a full 100% air gap between the two existing already separate systems… and other stupid stuff like that.

65535 • April 22, 2015 3:50 AM

@ Ken

“…there are TLA lists for talking about technology, I would expect I have been on them for a long time.”

Probably. I would prefer to stay off of government lists. But, what-the-hay.

It appears the main sell point of “Integrated Modular Avionics” is Costs [including maintenance costs]. That doesn’t inspire a lot of confidence.

It looks like safety takes a backseat to cost – but who knows.

[From your A 380 pdf]

“Why IMA?

“The Indirect Consequences

“Every system = 1 or more computers / controllers

“Every aircraft type = new computers

“Every computer = Airframer development and management costs

“Part number costs

“Documentation

“More wires

“More power

“This implies that quantities of maintenance spares be stored for each fleet at different places.

“During the aircraft life cycle, the cost of modifications, including parts obsolescence Mitigation and functional upgrades, becomes even more significant for the airlines…”

See pdf page 4 or 5:
http://www.artist-embedded.org/docs/Events/2007/IMA/Slides/ARTIST2_IMA_Itier.pdf

It does appear that cost is the main selling point – which may or may not increase safety. That is somewhat troubling.

fajensen • April 22, 2015 4:44 AM

@Ken

A quick Google on “ARINC 664” came up with a set of 55 slides “AFDX_Training_October_2010_Full.pdf” (I refuse to post the 3 kB Google Link).

Slide 7 says:

“””
AFDX Communication protocols have been derived from commercial standards
IEEE802.3 Ethernet MAC addressing
Internet Protocol IP
User Datagram Protocol UDP
SNMP
ICMP
Provisions have been added to ensure Deterministic Behaviour. End-Systems perform traffic shaping which is enforced by Switches. Switches perform traffic policing and static routing of frames.
“””

UhhHuh – SNMP is a toxic plague from hell that must be cleansed with nuclear fire!
They would’t be doing any network management with that, now, would they. Really?!

Slide 12:
“””
Avionics applications residing at End-Systems exchange messages via the services of the User Datagram Protocol Layer (UDP, Layer 4) with underlying Internet Protocol (IP, Layer 3)
The UDP Protocol is also the base for upper layer protocols for maintenance purposes (SNMP Simple Network Management Protocol) and File Transfer Services (TFTP Trivial File Transfer Protocol)
End Systems also have to support ICMP Internet Control Message Protocol based on IP protocol, but still residing in Layer 3. (known as “Ping command”, “Echo Request/Reply” on ICMP)
AFDX Switching is based on the MAC Destination Address (Layer 2)
AFDX provisions for deterministic are implemented on Layer 2 only
“””
NOOOOO! and TFTP … For What?!

The switches do “traffic shaping” – meaning, that they may drop packages or they will buffer? Seems they can drop and “no data” means “no frames sent” (slide 12), one could, perhaps, copy packages and re-inject them, slowly increasing the flow rate to swamp the real data?

They allow IP fragmentation (slide 15) “The End System has to maintain the Frame size for all transmitted frames with respect to the configured max. Frame Size for each VL e.g. IP Fragmentation” – so they allow DOS’ing too.

There is no mentioning of cryptography. One would therefore assume that if one can get the right bits juggled together, this frame will be valid. Most “hacking” tools can do raw frame-formatting.

I see a HUGE attack surface presented here if anyone can get on that network – especially when considering the ponderous aviation HW-certification procedures; some of this code might be years older than the exploit tools.

I sincerely hope that “people in charge” at least regularly let some hackers on a plane to have a go at it for some nice prices good enough to make them sign an NDA – similar to what Google does.

Sigh. My old fear of flying has returned.

wiredog • April 22, 2015 5:01 AM

Why is everyone posting on this having an attack of the stupids? Imagine this scenario:
Aircraft is cruising along, Bad Guy hacks into the control system via the in aircraft wifi. Pilot then physically pulls the breaker on the wifi. Emergency over. It’s not a movie plot threat, it’s a half hour sitcom threat.

arnim • April 22, 2015 5:17 AM

“Newer planes such as the Boeing 787 Dreamliner and the Airbus A350 and A380 have a single network that is used both by pilots to fly the plane and passengers for their Wi-Fi connections.”

That’s at least not true for the A380. There’s no software firewall between cockpit and cabin but they left out the 2 receive wires from the twisted pair cable on the switch. That’s a hardware barrier and allows traffic only in the direction from cockpit to cabin (e.g. broadcast of position and flight data for the in-flight entertainment systems).

Source: c’t magazine 17/2005, page 84ff (Sorry, German only, http://epaper.heise.de/download/archiv/3774167dd846/ct.05.17.084-091.pdf)

Martin • April 22, 2015 5:30 AM

The URL included tells your blog that I am coming from Twitter “…?utm_source=twitterfeed&utm_medium=twitter” even though I come from G+. No wonder everyone thinks G+ is a ghost town 😉

keiner • April 22, 2015 5:38 AM

Wouldn’t be a psychiatric diagnosis of the pilot be the method of choice to cover-up such a plot?

Just asking…

Clive Robinson • April 22, 2015 7:23 AM

@ Bruce,

In February, Toyota recalled 1.9 million Prius cars because of a software vulnerability.

You have missed mentioning the likely result of “recalls” on products is to make future products even more insecure.

I’ve no idea how much this particular recall cost Toyota directly and indirectly, but it’s likely to be over a billion USD.

Such “returns costs” can and have wiped companies out of existance, and is the reason why “Quality Control / Assurance” championed by the British Standards Institute is in existance and now championed by most international and national standards bodies and industry organisations and associations.

You can be sure that the Toyota recall has been seen and discussed by many senior execs in many manufacturing organisations and they will have looked at ways of mitigating similar costs.

And it is almost guaranteed that the “race for the bottom” soloution will have been selected, with the attendent vastly increased reduction in security.

That is they will in all probability go for the “push patch” solution as the software industry has sunk to.

However as Stuxnet and other attacks have shown such systems are increadibly fragile security wise and rely entirely on the proven to be vulnerable to multiple attack vector “code signing”.

From a “exec level” view “code signing” is by far the least expensive way to go, and you can be fairly certain that unless there is a way to “externalise costs” onto others this is the way they will probably go.

The way they should go which is to improve the software process is probably the most expensive route, and will not be taken unless there is legislation in place to force it. We have seen this in the past not just with “Lemon Laws” but going all the way back to Victorian boiler artificers, that gave rise to “engineering”.

One thing that this calamitus historical past does show is that contrary to what many always claim as being ‘a burden on industry that will damage it beyond recovery’ etc, such legislation has actually produced significant and very profitable inovation that has driven industry forward.

Andy • April 22, 2015 7:27 AM

There is absolutely no need for broadcasting position or flight data for the in-flight entertainment systems.

Just setup a LCD screen connected to the secure network showing the desired information and a decent camera connected to the unsecured network recording the LCD screen. Enclose the duo in a box for better protection against dust and undesired light reflections.

You don’t need UHD for this, just an idea of where the plane is.

Clive Robinson • April 22, 2015 7:33 AM

@ Andrew Wallace,

This discussion should ideally be done in recognised industry groups with vetting in place.

That is a very silly thing to say.

History shows that such setups are not only counterproductive the easily get “captured” by vested interests that want almost entirely the opposite of what the public need and expect.

Also it would significantly aid those who would seek to do harm, and not in any way hinder them.

As I’ve said in the past your comments destroy any credability you might once have had in your claims of being a security practitioner.

Archon • April 22, 2015 7:41 AM

“A terrorist hacks into a commercial airplane from the ground, takes over the controls from the pilots and flies the plane into the ground.”

You want odd? That was the plot of the series finale of “Martial Law”, PRE-9/11. In a morbidly amusing twist, they were flying the first plane(s?) into parts of New York City. The episodes got pulled from the reruns Spike (or was it still TNN then?) was running for obvious reasons, even though the episode had a… quaint… understanding of terrorist plane hijackings. (For instance, there was no understanding that specific parts of NYC could be targeted rather than just “the city”.)

So what I’m basically saying is that the GAO is less inspired than 20 year old CBS TV shows.

Paul Marten • April 22, 2015 9:14 AM

Re Ken “It is highly unlikely, if that patent made it into product, that it is on Boeing’s commercial aircraft.”

Right, not on commercial aircraft, no. To port such a system would require procurement in onesies and twosies and slow, intensive testing. And who does that?

Someone with access to the patent-holders’ billing data by contract could unearth some fascinating patterns. Let the whistleblowing begin.

Mr. Sunshine • April 22, 2015 9:17 AM

  1. Actual sensible security measures are suggested.
  2. Money for implementation instead used for executive bonuses.
  3. BOOM.
  4. “Here at DeathscreamAir, we take our commitment to security seriously.”

  5. Ad blitz with megajoyjoyous people and a constantly climbing (never-ever-EVER descending) plane.

Late-Stage Pseudo-Capitalism. What could possibly go wrong?

fajensen • April 22, 2015 10:34 AM

Bad Guy hacks into the control system via the in aircraft wifi. Pilot then physically pulls the breaker on the wifi. Emergency over.

… off course this happens before the evil hax0rs manage to reconfigure the switches via SNMP or gets something to suck in a new OS via the TFTP/BOOTP option the testers forgot to switch off?

Vintage CISCO kit can be put out of action for up to 20 minutes just by firing off a “reload” command (restarts the device, load configuration from memory).

Flight qualified kit ought to be much better that that, but … is it?

Ralph Hartley • April 22, 2015 10:38 AM

Is there any evidence that MH370 was not hacked?

The worst case scenario is that it was a test of the attack, and the remote location where it apparently crashed was intended to bury the evidence, preventing the vulnerabilities used from being identified and fixed.

There are about a thousand 777s currently in service, about half of which are in the air at any one time. If all, or a substantial fraction, of those were hijacked at once and directed at buildings, that would be one of the worst cyber-attacks I can imagine.

Nick P • April 22, 2015 11:00 AM

@ Ken

I’m not affiliated with any of those programs. I just stay up to date on what the field is doing and develop stuff myself. Might be a nice job, though.

“The capabilities are being built, but not always being deployed. ”

The problem in a nutshell across all of IT.

@ Archon

You want odd? In this episode of Lone Gunmen, a U.S. government group tries to crash an airliner into the World Trade Center to spur arms sales. That aired in March 2001. Best TV coincidence ever.

dvv • April 22, 2015 11:15 AM

@Andy, and for Internet and cell phone connectivity, there will be a couple of FBI-vetted and NSA-trained TSA signalmen to fill the air gap.

BTW, I use an app called “Avare” on my phone, and it works pretty well displaying my position on aviation charts even from inside a commercial airliner.

rgaff • April 22, 2015 11:28 AM

@Andrew Wallace

Why are you still here? Shouldn’t you be in prison already, for saying publicly how to hack a plane? At least, according to your other posts that’s where you apparently should be…

ConcernedCitizen • April 22, 2015 1:12 PM

@1 According to Ken, the reason the network is either single (A350, A380, B787) or the IFE/WiFi network is connected to the avionics/Satcomm network is that they require shared resources which are expensive (in terms or weight, space and maintenance) to duplicate. Specifically the interface the outside world – transmitter/receiver and antenna array.

Nick P • April 22, 2015 2:01 PM

@ ConcernedCitizen

The point of IME is lower cost, greater flexibility, easier maintenance, and improved performance with use of better tech. Why have two satellite networks when you can safely multiplex two networks over one link? Why rely on 100+ computers to work right when you can do 3-20?

A lot of what they’re doing makes sense. It might increase risk in some areas and reduce it in others. Good news is they’re using rigorous development and evaluation methods for a lot of this stuff. I expect that market to keep driving quality up and cost down for some time. I’ve often recommended them for reliability-oriented commercial systems.

So, what we need now is for domain experts and security engineers to do a thorough analysis of the whole thing. ATC, IME, the networks, hardware, emanation issues… all of it. Then we know the risks and can start dealing with them incrementally. Maybe even knock out many in parallel given the specialization and continuous improvement that market displays.

Hans • April 22, 2015 2:16 PM

I can confirm what @arnim said above about the one-way electrical isolation of the flight-deck network on the A380. They called it the “ether-diode”. I can’t say if they’ve maintained that level of isolation to this day, because it definitely limited what you could “do” as far as sexy onboard applications. But they were pretty serious about it up until the first delivery, which is about when I dropped out of that circus. One wonders what the B787 or the A350 have done.

I worked at a startup doing in-flight enterntainment (IFE) stuff back in the 2000’s. We worked closely with Airbus during the development of the A380 (and eventually became a partly-owned subsidiary). I spent far too many hours in Hamburg and Toulouse working with with their onboard network people. While I remember a bunch of their decisions were half-assed, one thing they one thing they were petrified of was someone in the cabin or on the ground getting access to the flight-deck systems via the network. This was after all “post 9/11” (barf).

I will say that working with Airbus (I worked with Boeing too, but not as much) made me hope to never fly on an Airbus plane. Something about seeing how the sausage is made….

Having read a few good GAO reports, I have to say that this one is pretty lousy, particularly when they drifted into aircraft network security. It’s made for great headlines, but you should notice that they don’t seem to have talked to any aircraft builders (e.g. Boeing, Airbus), only to some people at their suppliers (e.g. Rockwell). My guess is that the latter likely want anything that will increase spending, so a crisis or movie plot is always good for that.

brucella, lol • April 22, 2015 2:20 PM

“Today’s top-secret National Security Agency programs become tomorrow’s Ph.D. theses and the next day’s hacker’s tools.”

no, it would be more accurate to say:

“Today’s top-secret National Security Agency programs and hacker tools become tomorrow’s Ph.D. theses.”

because hackers aren’t necessarily behind the NSA in terms of development of capabilities

name.withheld.for.obvious.reasons • April 22, 2015 3:54 PM

@ Bruce Schneier

I believe that we have to step back from the technological issues (there are too many to sufficiently address the implications and impacts) and address the societal direction and rational that is used to “plug everything in” product approach that is so popular in industry and an insecure society (I believe the U.S. public is completely compromised). When will it stop…does it need to stop…is there a technological curve that a some point becomes orthogonal to the function of society itself? We are head long in this “plug everything in” parade and we’ve not asked the prerequisite questions which is part of a civil, democratic (supposedly), society. It is unwise to embrace the comfort of a ride across the river on the back of a crocodile without considering what it might cost.

When will the mindset of “plug everything in” become co-terminus with reason…after the last IoT butt-plug?

It is far too easy to, as even a hobbyist, to contemplate how to subvert, deny, destroy, or disable any technological system. We are continuously exercising tactical methods (how is this or that weak, insecure, or whatever) without any described mission, vision, or strategy (Why do I need to plug my toaster into the Internet, or, Everything will be plugged into the network or it will not function)…

Andrew Wallace • April 22, 2015 5:04 PM

rgaff,

There is a difference between premeditated distribution and publication of unpatched technical software vulnerabilities to organised criminals and me pointing out Boeing have an inflight diagnostic suite to send data to engineering teams on the ground.

Also, in the previous thread a male had tweeted a perceived threat while he was on an airliner full of passengers and was detained by the authorities and banned from the entire United fleet.

A completely different kettle of fish from anything I have mentioned here and likely ever to mention in a public place where organised criminals lurk.

And while you criticise the use of vetted industry groups. You are much more likely to get a high quality discussion from credible experts as people feel more at ease talking about possible vulnerability vectors.

Andrew

PatG • April 22, 2015 8:32 PM

I was sitting next a 787 captain on a flight home last week and we talked at length about a bunch of stuff. One of the things he mentioned to me was how crappy those IFE systems are. Apparently they regularly overheat and in one instance one of them was hot wired with no circuit breakers and when it overheated it started a fire. I would be more worried about sabotage in this manner than the possibility of hacking into the flight control systems as it seems it would be much easier.

Wesley Parish • April 23, 2015 4:01 AM

It seems to me, after reading all these comments, that a certain amount of obfuscation has taken place.

The severing of the receive wires in the ethernet cables seems an excellent idea.

I’d go further: why not make the flight control network a fibre network, and the passenger network a copper? You’d have to multiplex the transceivers, at least for air-to-ground Internet, and that would be the major vulnerability, so you’d work on hardening those. Unless you multiplexed the satellite transceivers and went through that way. But the rest of the network could be (relatively) safely isolated from each other: after all, the passenger-side network’s not doing anywhere near as much or as critical as the aircrew-side network, so it can be handed over to a (legacy?) copper network while being doubly isolated, first through a one-way optical-electric connectors and then the “ethernet diode” aforementioned.

Mike (just plain Mike) • April 23, 2015 2:03 PM

@Ken – this is a bit late I know – you’re probably not here any more – but, regarding the practicalities of remote control you say:

“I am highly skeptical of claims of being able to remotely control (crash) an airplane from the ground”

and then you say

“The worst case scenario I have seen amongst my aviation colleagues is a corrupted flight plan sent to the flight deck, accepted without review by the pilots, and then having the pilots not notice they aren’t going where they thought”

Are you saying a new flight plan can be transmitted from the ground (by those authorised/authenticated to do so), unsolicited by the crew, and that it will then be ‘presented’ to the pilots for review – and if they do the equivalent of ‘clicking OK’ then it will be automatically entered into the auto-pilot?

I’m thinking, if this is so, then a back-door (specifically designed into the software of the aircraft’s systems – I’m not talking about hacking here) could perhaps recognise ‘specially crafted’ flight plans, skip the human-review stage, automatically re-programme the auto-pilot and then also trigger code in other systems to lock the pilots out of controlling the aircraft. You wouldn’t be able to land the thing, but you could send it off somewhere out of the way, and you’d be denying control to hijackers in a 9/11 scenario in which everyone on board was going to die anyway. I’m thinking in the light of your remarks that this might just require software changes (perhaps in wake of post 9/11 concerns). I’m asking only if you think that such a thing is plausible – personally I think it would be a very bad idea, and I imagine people in the industry would agree – but I think quite a lot of other people might think it was a good idea and would be confident they could keep the secret of how to craft the flight plans to trigger it (final destination some special way-point, or The South Pole or something like that perhaps).

rgaff • April 23, 2015 5:49 PM

@ Andrew Wallace

No no no… that other thread was about some idiot on a plane tweeting something similar to “lessee… so many vulnerable systems on this plane, there’s XXX… and YYY… and ZZZ… which one should I hack today… har har har” which is obviously a joke, and not in good taste. And the FBI KNEW this, and knew that he was not a real threat, as evidenced by them letting him land at his normal destination before taking him in…

But instead, YOU took it as “this guy just publicly released vulnerabilities to criminals, [by merely mentioning what their acronyms were!] he should be put in prison”… and you spent that whole comment section arguing that point about it.

Then you come here to this thread, and YOU YOURSELF mention a vulnerability AND you specifically SAY how YOU WOULD use it to hack a plane. What’s good for the goose is good for the gander. Sorry mate. Do not pass “go” do not collect $200.

Rather, both you and that tweeter are idiots, and just being an idiot isn’t a crime in this country… yet… unless people like you have your way anyway. This is my main point about this whole thing. So you might want to reconsider your position, since your own position puts you in the wrong just as much.

moo • April 23, 2015 7:21 PM

@wiredog:

“Why is everyone posting on this having an attack of the stupids? Imagine this scenario:
Aircraft is cruising along, Bad Guy hacks into the control system via the in aircraft wifi. Pilot then physically pulls the breaker on the wifi.”

That would only work against a Bad Guy who was trying to actively control the plane somehow over wifi. Suppose he hacked in and deployed some malware with a pre-programmed behavior, and then activated it. The pilot/passengers would then notice that something is amiss but by then its too late to do anything about it.

rgaff • April 23, 2015 8:33 PM

@moo what? you mean hackers don’t carry around virtual reality goggles and gloves everywhere to remote control everything? that’s how they do it in the movies with their little hippie mobile hacking vans!

Benni • April 24, 2015 12:33 AM

In that connection this car hack may be interesting:
http://www.heise.de/newsticker/meldung/ConnectedDrive-Der-BMW-Hack-im-Detail-2540786.html

All BMW cars nowadays have a connected drive feature. This can be used to open a locked car remotely. All BMW cars have the same cryptographic keys in connected drive. And the communication was, until recently, not encrypted. So, a few weeks ago, a low skilled hacker could open all BMW cars remotely if he wanted.

BMW now has made an update. This was transmitted wireless. The update was that BMW cars will now use SSL in connected drive….. given how easy SSL can be hacked, there is a large possibility that someone at NSA still has access to all BMW cars and can unlock them if he wanted.

Andrew Wallace • April 24, 2015 7:33 AM

rgaff,

I do believe aircraft sending a flow of diagnostic data through the air to ground teams could pose a vulnerability if sophisticated hackers are able to intercept it.

GCHQ BUDE facility in South West England is able to do this with ease. Therefore sophisticated hackers could possibly hoover up the same data.

Andrew

rgaff • April 24, 2015 3:17 PM

@ Andrew Wallace

I agree that having access to diagnostic data could probably be used to help craft further attacks more effectively, therefore it should be hidden. However this is not my point.

My point is, you saying “if I were to hack a plane, I’d do it like XYZ” gives just as much information to the bad guys about how to hack planes as the tweeter guy saying “look at XYZ on this plane, so easy to attack it’s silly” so you condemning him for that is also condemning yourself. You are the same as him in this respect. You’re both sharing information with bad guys how to hack planes. If he must go to prison for it, so must you.

TomTrottier • April 24, 2015 4:38 PM

If security were baked into hardware in a way which could not be duplicated or examined, perhaps one country could have a secure infrastructure by not allowing exports.

SchneieronSecurityFan • April 24, 2015 4:39 PM

“Newer planes such as the Boeing 787 Dreamliner and the Airbus A350 and A380 have a single network that is used both by pilots to fly the plane and passengers for their Wi-Fi connections. ”

“Previous planes had separate networks, which is much more secure.”

Reminds me of the internet vulnerabilities with SCADA systems.

Utility companies used to lease their data links from long-distance companies or maintain their own telecommunication networks. A major vulnerability developed when the data communications was put onto the general internet. It was done in order to save money.

“

roger • April 27, 2015 9:21 PM

The flight software in the Boeing, and I would guess Airbus, is compliant with RTCA DO-178B. Any attempt to cross over the application space should lead to an abort (SIGABRT) of the in-flight entertainment system as long as underlying UNIX real time system is not compromised.

Nathanael • April 29, 2015 9:46 AM

I’ve thought for several years that the NSA director, Director of National Intelligence, etc., need to be arrested on charges of endangering national security. Their efforts to make networks insecure have severely endangered the United States.

Nathanael • April 29, 2015 9:49 AM

If the in-flight entertainment network needs to know information from the avionics but not vice versa, there’s a known technical solution: the one-way feed. No information goes into the avionics, not even “requests” — the avionics just sends out a constant feed which can be monitored by others.

John Galt III • May 16, 2015 8:51 PM

apparently there is an update to the situation

http://www.wired.com/2015/05/feds-say-banned-researcher-commandeered-plane/

enough of the FBI are dirty that some skepticism is warranted. The FBI in Boston let Whitey Bulger continue his murder-for-hire enterprise, for years, under their watchful eyes, in return for information.

The FBI agent who shot Todashev during interrogation had a colorful history before his FBI employment:

http://whowhatwhy.com/2014/05/17/todashevs-killer-no-wonder-his-identify-was-secret/
…
He also took the Fifth Amendment and later testified under immunity during a corruption investigation into a rogue police unit called “The Riders” whose members were charged with making false arrests, planting evidence, and falsifying police reports. The city settled the federal lawsuit for $10.9 million. McFarlane wasn’t charged in that case, or in three other internal affairs investigations, although a prosecutor accused him of being misleading.
…
McFarlane retired from the Oakland Police Department in 2004 on medical disability after repeatedly injuring his leg and breaking his ankle, securing a lifetime $52,000-a-year pension. Four years later he joined the FBI, raising questions about how he passed both the rigorous background check and the FBI’s physical requirements.

Sancho_P • May 17, 2015 6:35 AM

I know I’m far away, far away from reality.
However:
What I understand the whole story calls for jail time of top figures at Boeing and Airbus, and probably from the FBI.
For years they knew about serious issues but ignored them.

Our standard reaction: Shoot the messenger, lie, deny, hide it under the carpet.

vesperto • May 18, 2015 1:56 PM

“Newer planes such as the Boeing 787 Dreamliner and the Airbus A350 and A380 have a single network that is used both by pilots to fly the plane and passengers for their Wi-Fi connections.”

Ah yes, when management decisions overrule engineering decisions.

Gray • June 1, 2015 5:44 PM

something germane to the discussion: the nav systems I’m told are based on green hill’s proprietary integrity kernel: http://www.ghs.com/products/rt…
This kernel has achieved some pretty amazing safety and security certifications:
FAA: DO-178B, Level A (INTEGRITY-178 RTOS)
NSA: EAL 6+ High Robustness Common Criteria
That its not windows or linux based essentially drops the risk probability of the flight systems being hijacked/hacked thru the wifi service substantially – if not all the way to zero.
The outcome of the risk assessment swings radically on this one bit – i’d say, most critical bit – of information (Risk and Probability plummets as Vulnerabilities and Threats both fall to zero; r=pvta) yet no one seems to have loudly pointed this out anywhere online or in the recent hearings, and that includes Wired, GAO, Boeing and Airbus.
The primary takeaway points out the importance of the risk assessment process and thinking in terms of probabilities. We risk wasting precious time & resources when the discussion occurs outside of this framework.

name.withheld.for.obvious.reasons • June 1, 2015 11:48 PM

@ Gary

The primary takeaway points out the importance of the risk assessment process and thinking in terms of probabilities. We risk wasting precious time & resources when the discussion occurs outside of this framework.

Two issues; first is the robustness of this discussion may be limited to the level of comfort others feel in expressing what the true risks are…if you haven’t noticed, the presence of an all encompassing surveillance state has the tendency to produce self censorship. This topic falls right along these lines, so I could see the reticence to make statements on public blogs that might have one fall afoul of the surveillance state.

Second point, multiple layers in the avionics (flight, nav, control, comm) systems provide for multiple targets when it comes to hardware, OS, and/or application layer attacks. Being familiar with aerospace systems (okay, maybe more than familiar) it is obvious that when the field of view is extended to the “System of Systems” scope, the problem space that includes attack vectors (or surface area) is logarithmic in scale when that space is evaluated as system components and not as a “System of Systems” architecture.

As a post script, the fact that any OS qualified as DO-178B/EAL+7 does not belie the truth concerning “out-of-band” and “out-of-bounds” conditions that often lead to component failure (of the soft type). But more concerning is the “embedding” of application level code closer to the hardware layer. I’ve seen decisions made to implement specific platform technologies in critical applications that would be considered less than robust…but because of time-to-market and familiarity with tool chains and sets…bad decisions were left to stand.

august medina • April 29, 2016 3:30 AM

Need a hacker? I was in a bit of a tough spot and didn’t know what to do until a friend recommended this guy Aviv Nadav at whitewebdemon@gmail.com. I was pleased with his professionalism and thoroughly satisfied with how he handled it. He’s your guy if you need one,does from facebook,gmail,yahoo,hotmail,school score upgrade,bank jobs,cloning phones,credit card hacks as well as blank atm’s,make as much as $20,000 dude’s kinda picky though so make sure of the reference.august referred you.You’re welcome

Name • November 22, 2016 9:17 PM

“Later in your essay, it sounds like you’re proposing a solution to the contradiction: Congress should change the NSA’s mission; at which time the NSA will wave their magic wand and make our software secure. I don’t believe their capabilities are so far ahead of ours that this is possible”
All they have to do is stop going out of their way to sabotage American companies as in Project BULLRUN, and stop bribing NIST to pass backdoored standards as in the Dual-EC fiasco.

If they actually started using their billions of dollars a year of taxpayer money to fix vulnerabilities in open source software and responsibly disclose vulnerabilities they find in closed software, their existence would be justified and they wouldn’t be guilty of ongoing treason.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.