Security Risks of Embedded Systems

We're at a crisis point now with regard to the security of embedded systems, where computing is embedded into the hardware itself -- as with the Internet of Things. These embedded computers are riddled with vulnerabilities, and there's no good way to patch them.

It's not unlike what happened in the mid-1990s, when the insecurity of personal computers was reaching crisis levels. Software and operating systems were riddled with security vulnerabilities, and there was no good way to patch them. Companies were trying to keep vulnerabilities secret, and not releasing security updates quickly. And when updates were released, it was hard -- if not impossible -- to get users to install them. This has changed over the past twenty years, due to a combination of full disclosure -- publishing vulnerabilities to force companies to issue patches quicker -- and automatic updates: automating the process of installing updates on users' computers. The results aren't perfect, but they're much better than ever before.

But this time the problem is much worse, because the world is different: All of these devices are connected to the Internet. The computers in our routers and modems are much more powerful than the PCs of the mid-1990s, and the Internet of Things will put computers into all sorts of consumer devices. The industries producing these devices are even less capable of fixing the problem than the PC and software industries were.

If we don't solve this soon, we're in for a security disaster as hackers figure out that it's easier to hack routers than computers. At a recent Def Con, a researcher looked at thirty home routers and broke into half of them -- including some of the most popular and common brands.

To understand the problem, you need to understand the embedded systems market.

Typically, these systems are powered by specialized computer chips made by companies such as Broadcom, Qualcomm, and Marvell. These chips are cheap, and the profit margins slim. Aside from price, the way the manufacturers differentiate themselves from each other is by features and bandwidth. They typically put a version of the Linux operating system onto the chips, as well as a bunch of other open-source and proprietary components and drivers. They do as little engineering as possible before shipping, and there's little incentive to update their "board support package" until absolutely necessary.

The system manufacturers -- usually original device manufacturers (ODMs) who often don't get their brand name on the finished product -- choose a chip based on price and features, and then build a router, server, or whatever. They don't do a lot of engineering, either. The brand-name company on the box may add a user interface and maybe some new features, make sure everything works, and they're done, too.

The problem with this process is that no one entity has any incentive, expertise, or even ability to patch the software once it's shipped. The chip manufacturer is busy shipping the next version of the chip, and the ODM is busy upgrading its product to work with this next chip. Maintaining the older chips and products just isn't a priority.

And the software is old, even when the device is new. For example, one survey of common home routers found that the software components were four to five years older than the device. The minimum age of the Linux operating system was four years. The minimum age of the Samba file system software: six years. They may have had all the security patches applied, but most likely not. No one has that job. Some of the components are so old that they're no longer being patched. This patching is especially important because security vulnerabilities are found "more easily" as systems age.

To make matters worse, it's often impossible to patch the software or upgrade the components to the latest version. Often, the complete source code isn't available. Yes, they'll have the source code to Linux and any other open-source components. But many of the device drivers and other components are just "binary blobs" -- no source code at all. That's the most pernicious part of the problem: No one can possibly patch code that's just binary.

Even when a patch is possible, it's rarely applied. Users usually have to manually download and install relevant patches. But since users never get alerted about security updates, and don't have the expertise to manually administer these devices, it doesn't happen. Sometimes the ISPs have the ability to remotely patch routers and modems, but this is also rare.

The result is hundreds of millions of devices that have been sitting on the Internet, unpatched and insecure, for the last five to ten years.

Hackers are starting to notice. Malware DNS Changer attacks home routers as well as computers. In Brazil, 4.5 million DSL routers were compromised for purposes of financial fraud. Last month, Symantec reported on a Linux worm that targets routers, cameras, and other embedded devices.

This is only the beginning. All it will take is some easy-to-use hacker tools for the script kiddies to get into the game.

And the Internet of Things will only make this problem worse, as the Internet -- as well as our homes and bodies -- becomes flooded with new embedded devices that will be equally poorly maintained and unpatchable. But routers and modems pose a particular problem, because they're: (1) between users and the Internet, so turning them off is increasingly not an option; (2) more powerful and more general in function than other embedded devices; (3) the one 24/7 computing device in the house, and are a natural place for lots of new features.

We were here before with personal computers, and we fixed the problem. But disclosing vulnerabilities in an effort to force vendors to fix the problem won't work the same way as with embedded systems. The last time, the problem was computers, ones mostly not connected to the Internet, and slow-spreading viruses. The scale is different today: more devices, more vulnerability, viruses spreading faster on the Internet, and less technical expertise on both the vendor and the user sides. Plus vulnerabilities that are impossible to patch.

Combine full function with lack of updates, add in a pernicious market dynamic that has inhibited updates and prevented anyone else from updating, and we have an incipient disaster in front of us. It's just a matter of when.

We simply have to fix this. We have to put pressure on embedded system vendors to design their systems better. We need open-source driver software -- no more binary blobs! -- so third-party vendors and ISPs can provide security tools and software updates for as long as the device is in use. We need automatic update mechanisms to ensure they get installed.

The economic incentives point to large ISPs as the driver for change. Whether they're to blame or not, the ISPs are the ones who get the service calls for crashes. They often have to send users new hardware because it's the only way to update a router or modem, and that can easily cost a year's worth of profit from that customer. This problem is only going to get worse, and more expensive. Paying the cost up front for better embedded systems is much cheaper than paying the costs of the resultant security disasters.

This essay originally appeared on Wired.com.

Posted on January 9, 2014 at 6:33 AM • 75 Comments

Comments

RichJanuary 9, 2014 7:10 AM

Maybe that $300.00 Linksys WRT54G re-make isn't so expensive after all. Unfortunately most people won't see that before their home networks are riddled with worms.

NeilJanuary 9, 2014 7:14 AM

So, what about options like pfsense.org? Is that a good way of addressing the issue for the individual/small business?

IanJanuary 9, 2014 7:20 AM

Paying the cost up front for better embedded systems is much cheaper than paying the costs of the resultant security disasters.
Your average consumer is just not going to pay extra for some nebulous security benefit that cannot be demonstrated. How can he tell that the more expensive router really does have all the latest patches applied and can be updated to prevent hacker access. As an embedded device programmer for the last 30 odd years I know I couldn't.

I can just imagine trying to justify to my wife that we should buy the more expensive gadget rather than buy the cheaper one and have a night out at the theatre. Not a happy picture and, as she often says, "You can only spend each pound note once."

hihihiJanuary 9, 2014 7:21 AM

the best option for soho users is not pfsense (which requires half-decent dedicated hardware such as an alix board) it is open-wrt, dd-wrt etc (which has support for most home routers). anyone with moderate tech skills can flash open-wrt onto a £20/$30 router in

http://wiki.openwrt.org/toh/start

Peter GalbavyJanuary 9, 2014 7:32 AM

As we move onto more and more life critical systems either in the home/car/office or attached to the user there will need to be more disclosure much like the calls for opening voting machinery code to inspection. There will have to be a way for independent scrutineers to review this stuff before it becomes dangerous.

Unless politicians, in their guise as law makers, get involved soon this is unlikely to happen soon enough and the shit will truly hit the fan - if not literally then certainly figuratively in the not so distant future.

Anderer GregorJanuary 9, 2014 7:43 AM

What users have learned with smartphones, and what they are starting to learn with routers, is: Updates break things. Software, tethering, etc for the former, and the last couple of updates our ISP installed on our cable modem broke, in that order of subsequent updates, the answering machine, SIP, and IPv4 (and with it the capability to connect to our network from IPv4 machines). So unless ISPs/telcos/manufacturers learn not to remove capabilities from their products that their customers held dear -- even if this may drive them into buying newer products -- these customers have a very strong incentive not to install any updates.

DavidJanuary 9, 2014 7:44 AM

As far as home routers, it's pretty difficult position for the consumer because one you've bought the unit of the shelf (with no current status of installed firmware version vs latest firmware version), you've finished your financial transaction with the manufacturer. They stand to make not a penny more to spend time in r&d to produce patches or fixes.

Rick LobrechtJanuary 9, 2014 8:23 AM

Not to sound like a fanboy, but this is one of the main reasons why I went with Apple networking hardware. They do push firmware updates, and you are notified in a way that you notice (the update comes along with the updates for your Mac.) Yes, the hardware is expensive, but less expensive than the new Linux Linksys router mentioned above.

cpragmanJanuary 9, 2014 8:24 AM

And now manufacturers of durable goods (TVs, refrigerators, cars, etc.) are trying to differentiate their products form the competition by embedding computers in their appliances. The reason seems to be more about differentiation from the competition than providing an actual benefit to the consumer. Face it, PCs and routers aren't "durable goods". Even if they get compromised, they will get replaced in a few years anyway, so they will get updated. Appliances don't get refresed on that sort of schedule.

As a consumer, I want something that will work reliably for years and years. Put an embedded PC in a fridge, why? That's just putting so many new failure modes into such a simple appliance.
Those things will be sitting out there for decades, still doing their malware thing, whether I still own it, or have passed it on to my kids. How long will that fancy software even be relevant?
I already own a TV that requires a proprietary interface that only an authorized rep can use to install updates. The process is so cumbersome, I doubt anybody has ever tried doing it. I shiver to think about having a similarly unfriendly system in every appliance.

AndrewJanuary 9, 2014 8:24 AM

Here's a magic word for you guys: Cloud!

If you believe the marketing/propaganda/public_excitement departments of most IT companies it's quite a wonderful thing. Yank the administrative tasks from the hands of the incompetent users and give them to the big brain experts in the GooglePlex.

Actually I know a big ISP 'round here that uses this scheme - end user routers are centrally managed by the company and end users have access to limited subset of customization options via the portal where they pay and manage their subscription plan.

Of course then comes that NSA/Snowden business that's been hyped lately... if you're willing to give an ear to those illegal and renounced by the world government as utter nonsense rumors.

DanJanuary 9, 2014 8:31 AM

Open Source has to be the way to go here. As long as the hardware is compatible with Open Sources packages such as DD-WRT et al, users can get the latest up-to-date patches along with vastly better service and reliability.

My last router took minutes to reboot after even the most trivial change. It crashed constantly. There was no incentive for the manufacturer to develop decent software for it. All they needed was enough to get it out the door so that it couldn't be returned.

So I flashed DD-WRT onto it. Suddenly everything's rosy, working fine, reliable, with features I never dreamed of. I can ssh into it. I can run a webserver on it. There's a stateful firewall. For crying out loud there are data usage graphs on the administration webpage.

This, more than anything, seems the way to go. We take reliability and security away from the folks who have no incentive to do more than a minimal job, and give it to the guys who have every incentive to do a good job. The original manufacturer benefits from the deal with less work for a better product. And users can update to their hearts content.

Clive RobinsonJanuary 9, 2014 8:40 AM

@ Bruce,

Your comments about the "embedded market" and the way it works is but one sector not the whole. Specificaly higher functioning non Real Time (RT) Fast Moving Consumer Electronics (FMCE).

It's important to note that "the rot" you describe actualy started with propriatory closed US mobile phone standards and the chip sets to support them back in the 1980's. Back then you had "to buy the licenced chip set and software" to manufacture a phone, there was no option. Open standards in this area started with the European Community, where "national fiefdems" were the norm, which actually resulted in the opposit effect than the likes of Politicians and Gov B'craps expected. It decimated home market manufacturing in favour of Far East manufactures, because of the "small national market" and the costs involved with "jumping through hoops" to meet individual national standards. Far East manufactures took the view that the would produce for all European National Markets, and set up "standards teams" the cost of which got spread across all national markets not just a single market. The European Economic Community realised this was a major major stumbling block and came up with common standards in a hierachy below a framework standard (see LV Directive and RT&TTE Directives).

As I've said many times befor the US is very bad at standards, specificaly NIST still have a "single national market" viewpoint subject to strong lobbying by major manufactures to have their patented technology included and thus have a nice little "licencing erner".

The solution for these FMCE "Internet of things" ills, is as I have stated before which is a framework of non proprietary / patent encombered standards. One part of which is to mandate a secure upgrade path that kills the "fall back" and "legacy support" issues dead.

As some will know (as they disagreed with me originaly) I've been banging on about this issue with NISTs myopic view point on this and other blogs for some years.

Further another important point is such regulation actually stops this chronic "race for the bottom" the "freemarket ideology" enforces.

As @ RobertT will no doubt point out one of the major reasons for "binary blobs" is to keep the chip internals hidden to avoide US Patent litigation. Open standards in a properly designed heirachy of standards that are not encombered with licenceable only patents is the only way forward to stop the issues you raise.

The "oft touted" argument that if companies could not patent and licence their technical inovations then they would not "inovate" is a compleat nonsense put forward by the lobyists of the likes of Apple. Companies will inovate if there is a perceived market need, open standards ease the inovation and actually make it considerably more profitable. The likes of Apple don't actually want to inovate they want market "lock in" for consumers and market "lock out" for other manufactures. It's got to the point where respected US Judges are saying "enough is enough" as they want rid of these extreamly harmfull and damaging "patent infringment" cases. Unfortunatly as normal they have an uphill struggle against incumbrants and their vested interest. Arguably the USPO review was way to little way to late, and unfortunatly US lobyists are hanging around the EU handing out the usual bribes, sexual coercion etc so that they can make the EU market not open but closed to US patent trolls.

Mike S.January 9, 2014 8:50 AM

A few weeks ago, my DSL modem/wireless router was reset out of the blue. I was already on the latest version of firmware available, but I wonder if someone used one of these vulnerabilities to get in and screw with it.

There's really no way for me to know if anything was left behind. As recent disclosures have shown, exploits that survive resets are commonplace these days.

Are there really any options available to people who care about security? I could buy a newer device, but there little chance a new one would not come pre-compromised and/or riddled with legacy vulnerabilities.

I don't think the ISPs will ever have a financial incentive or enough clout to change the situation. The person with a support problem today doesn't necessarily have the current version/model of hardware. The hardware vendor will say the problem is fixed in the current hardware, so they need to buy that. If they're already on the current version, they'll replace their hardware with one that's exactly the same and hope it doesn't happen again. Nobody has customer service who can analyze and work to resolve security vulnerabilities. There's no workable security upgrade model that doesn't include the customer paying for that customer service on an ongoing basis.

Driving towards open source components is a great idea. As long as someone has an interest in patching vulnerabilities and contributing them back to the community, everyone using that OS benefits.

paulJanuary 9, 2014 8:54 AM

(The other side of un-updated software is that at least you don't have a bunch of new features, each with its own holes to plug.)

Meanwhile, internet-of-things infrastructure makers are also leading a race to the bottom. One of the standard embedded-wifi chips for hobbyists and developers, for example, shipped with firmware that allowed only shortish (16-byte) plaintext passwords.

Malachi JonesJanuary 9, 2014 9:06 AM

The main issue here is the "Market for lemons" concept. This is that bad products can drive out good products when the buyers are unable perceive a difference between the two.

In the security arena, this means that it is rational for companies to deliver products with poor security features in order to maximize shareholder profit.

Steve MerrickJanuary 9, 2014 9:12 AM

We use Texas MSP430 and NXP LPC1700-series microcontrollers, and are delighted to be using hardware that can support C and a C compiler! We deal with FLASH memory sizes between 32 kBytes and 128 kBytes, with 1 to 32 kBytes of RAM. I'm not sure Linux would run in so small a space.

Your article does not accurately describe our business (Access Control) or our embedded products. I have no idea if our embedded systems are typical or not.

Brian M.January 9, 2014 9:24 AM

This last time around, I bought a Cisco ISA550. I was surprised to see that it can download and installs updates on its own. Of course I set that up immediately. That sort of functionality doesn't come cheap, though, so I'd never expect to see this in a cheap firewall appliance.

Jordan BrownJanuary 9, 2014 10:47 AM

My cheapo NetGear router downloads and installs upgrades. In fact, I went to look to confirm that and it tells me that an upgrade is available now.

How often does the vendor release an upgrade? Don't know.

Why isn't some of the answer holding the vendors liable for flaws in their products?

Some_Guy_In_A_DinerJanuary 9, 2014 10:48 AM

Embedded systems, including mobile phones, are in a horrible state of security. If I can't audit, patch, and update a device I CAN NOT trust the device.

qwertyuiopJanuary 9, 2014 11:28 AM

@Brian M - and you see that as a *GOOD* thing?

Maybe I'm paranoid (actually, no, I *AM* paranoid) so I'd like to know if there's an update and what its purpose is before *I* make the decision to install it or not. Given that it seems doubtful we can trust manufacturers any more then blindly accepting an update is a sure way to get the latest compromise installed.

Jason Richardson-WhiteJanuary 9, 2014 11:32 AM

A sequence of thoughts, part summary, part lead-in...

There's an interesting parallel in the energy sector: demand-side management. So-called "smart metering" could allow power distributors to structure costs dynamically --- on a house-by-house basis --- so as to pressure consumers to accept heating / cooling schemes which distribute power demand more evenly, leveling spikes due to so-called "peak" hours. Reduced maintenance cycles, lower "peak" power requirements, and much more, including lowering overall demand, roll out from these ideas.

They're nice ideas, but a major problem is securing the darn things. Apart from the obvious -- DDoS the recipient of the data from all those meters, thereby making power distributors blind and vulnerable -- it would be possible to hack the meters remotely, since they're connected to the internet.

But the differences with Bruce's post are illuminating. Because the power grid is *infrastructure*, it's subject to copious regulation. System architects will be paid (are being paid) lots of money to ensure that the system is... well, not "unhackable" but at least not a target for anyone but a foreign power. This will / would of course include regular patch management of end-point meters.

We can't regulate routers in the same way, of course.... or could we? Among the problems, short of a covert-channel hack, smart meters might tell relatively little than some aspects of a person's habits, but a router (as we now know) carries not only a person's actual communiques but *strongly* discloses their interests, habits, and much more. So no one wants the government getting involved, because the government isn't trustworthy -- they're provably backdooring routers.

But if we can't trust the market to secure embedded system and we can't trust the government, who can we trust...?

Some commenters (and Bruce) lean toward open-sourcing the code in routers (and other embedded devices). This would encourage security researchers (are there enough?) to publish vulnerabilities, creating pushback on manufacturers.

But, perusing responses above, we see (1) unless we assuage IP worries -- or do away with IP -- there is a strong disincentive not to give away source that could be used by big players (like Apple, like IBM, like Cisco) who have actual business units dedicated to nothing but patent revenue and (2) there aren't enough open-source hackers in the universe to create free open-source drivers for *every* device in the coming "internet of things".

Thinking about this... two solutions come to mind. First, do the impossible and make the political system work well enough to reform the IP system. (Bwa-ha-ha-ha... sorry, not laughing anymore). IMHO, it would take a 60s-like decade -- a generation of "protests" and "marches" that could make the 60's look positively calm. Maybe that's where we're headed (shrugs).

The second solution is just as radical but more passive -- strategic withdrawal. If you can't kill the beast, starve it. Let communities of hackers (in the polite sense) build the next generation of the internet, on open principles, and let the public in on it. (They're listening now, unlike in the 80's when ARPA-Net was a conversation in garages.) Do it on the theory, "if you build it, they will come."

The single largest technical problem is distributed DNS. It can't just be BitCoin, because BitCoin couples functions -- money and communication -- that we need to be able to decouple. The same principle goes for *content*. Letting Google buy Motorola...?!?! We need to de-couple content from its mode of distribution, so that the public can still get great content but do it on a *new* internet.

IMHO, distributed systems -- ones that no one has entirely under their control, not government, not even Google (lol) -- are the way to go. There are dangers, true, such as protecting bad / malicious content. But this is a frontier that needs to be conquered instead of being relegated to the bad guys.

JasonJanuary 9, 2014 11:33 AM

A sequence of thoughts, part summary, part lead-in...

There's an interesting parallel in the energy sector: demand-side management. So-called "smart metering" could allow power distributors to structure costs dynamically --- on a house-by-house basis --- so as to pressure consumers to accept heating / cooling schemes which distribute power demand more evenly, leveling spikes due to so-called "peak" hours. Reduced maintenance cycles, lower "peak" power requirements, and much more, including lowering overall demand, roll out from these ideas.

They're nice ideas, but a major problem is securing the darn things. Apart from the obvious -- DDoS the recipient of the data from all those meters, thereby making power distributors blind and vulnerable -- it would be possible to hack the meters remotely, since they're connected to the internet.

But the differences with Bruce's post are illuminating. Because the power grid is *infrastructure*, it's subject to copious regulation. System architects will be paid (are being paid) lots of money to ensure that the system is... well, not "unhackable" but at least not a target for anyone but a foreign power. This will / would of course include regular patch management of end-point meters.

We can't regulate routers in the same way, of course.... or could we? Among the problems, short of a covert-channel hack, smart meters might tell relatively little than some aspects of a person's habits, but a router (as we now know) carries not only a person's actual communiques but *strongly* discloses their interests, habits, and much more. So no one wants the government getting involved, because the government isn't trustworthy -- they're provably backdooring routers.

But if we can't trust the market to secure embedded system and we can't trust the government, who can we trust...?

Some commenters (and Bruce) lean toward open-sourcing the code in routers (and other embedded devices). This would encourage security researchers (are there enough?) to publish vulnerabilities, creating pushback on manufacturers.

But, perusing responses above, we see (1) unless we assuage IP worries -- or do away with IP -- there is a strong disincentive not to give away source that could be used by big players (like Apple, like IBM, like Cisco) who have actual business units dedicated to nothing but patent revenue and (2) there aren't enough open-source hackers in the universe to create free open-source drivers for *every* device in the coming "internet of things".

Thinking about this... two solutions come to mind. First, do the impossible and make the political system work well enough to reform the IP system. (Bwa-ha-ha-ha... sorry, not laughing anymore). IMHO, it would take a 60s-like decade -- a generation of "protests" and "marches" that could make the 60's look positively calm. Maybe that's where we're headed (shrugs).

The second solution is just as radical but more passive -- strategic withdrawal. If you can't kill the beast, starve it. Let communities of hackers (in the polite sense) build the next generation of the internet, on open principles, and let the public in on it. (They're listening now, unlike in the 80's when ARPA-Net was a conversation in garages.) Do it on the theory, "if you build it, they will come."

The single largest technical problem is distributed DNS. It can't just be BitCoin, because BitCoin couples functions -- money and communication -- that we need to be able to decouple. The same principle goes for *content*. Letting Google buy Motorola...?!?! We need to de-couple content from its mode of distribution, so that the public can still get great content but do it on a *new* internet.

IMHO, distributed systems -- ones that no one has entirely under their control, not government, not even Google (lol) -- are the way to go. There are dangers, true, such as protecting bad / malicious content. But this is a frontier that needs to be conquered instead of being relegated to the bad guys.

Tim van BeekJanuary 9, 2014 12:26 PM

This is a problem that car companies have handled for many years now. This is how it works at least for the ones that I know:

The responsible engineer at your car company writes the specification for embedded ECUs. This specification includes the mechanisms for software updates. Basically, these are certain services that every ECU needs to understand to perform updates of specified software components.

The car company provides the infrastructure needed to "talk" to a car to install a software update, which includes both hardware and software.

Each ECU provider is requested to deliver software updates with specific meta data to the car company.

The software updates are collected and distributed to every interested party, e.g. car workshops.

In effect, if you drive your car to specific car workshops, the mechanics will be able to determine if there is a software problem that has already been fixed and to apply the corresponding software update.

This has been working quite well for many years now. Of course, for cars, there are

detailed break down statistics,

usually a long guarantee period and

strong branding.

These and other factors put strong pressures on car companies to fix their software problems. But these factors seem to be almost nonexistent for ISPs. Has anybody heard of a "breakdown statistic" for ISPs that tells customers not to buy at company X because they will be hacked more often?

BTW, due to EU competition regulations, every car company has to provide both the hardware and the software needed to apply software updates to their cars. You can get all you need to update e.g. a BMW for approximately 5000$ (this is the legal path, the illegal is somewhat less expensive).

DanielJanuary 9, 2014 12:31 PM

A good article Bruce but it overlooks a key point. It's not only that these system are vulnerable but that such embedded system are protecting even more and more valuable stuff.

http://www.itsbaby.com/wearable-baby-monitor/

This is one of my favorite examples. Wait until hackers get their hands on that baby (pun intended). Let's see, they can hack the device to make the transmitter overheat and burn the baby (ha ha) or they could hack the device to show that the baby is dying or has a fever and freak the parents out and perhaps resulting in an unnecessary trip to the ER (ha ha). The are even more evil ideas I won't mention. And if society thinks hackers won't do that they have never heard of SWATTing. There are some hackers who just love to be mean.

So people need to sober up and think more about what they are place into the protection of these embedded systems.

Brian M.January 9, 2014 1:09 PM

@qwertyuiop:
Maybe I'm paranoid (actually, no, I *AM* paranoid) so I'd like to know if there's an update and what its purpose is before *I* make the decision to install it or not.

How do you know what is actually in the update, if you aren't compiling from source code that you have reviewed for yourself first? I doubt that any manufacturer is going to mention in the release notes that the router is now part of a government surveillance network.

I would really prefer that the equipment updates itself, or notifies me of an available update, say through SNMP. But the biggest problem is getting the manufacturer to create an update at all. For instance, I have a couple of cheap IP cameras for yard surveillance. I doubt that the manufacturer will ever create an update for those, no matter how many security holes are found in the firmware. What to do? I use a managed switch to put them on their own VLAN. Am I paranoid? No, I wanted to play around with VLAN permissions.

Remember, if the NSA gets your equipment first, all bets are off.

Andrew HamiltonJanuary 9, 2014 1:30 PM

No Software is "embedded" in hardware, at least not in a CPU.
The hardware always has to boot, start an os and load the software it is to run.
Which means that the OS/software can always be updated with patches, and ideally run security programs, although I suppose this is ott for a tv set cpu.
Intel acquired Mcafee specifically to enable 100% security by Mcafee products, which requires knowledge of the cpu microarchitectures to properly identify and resolve any security risks tey may enable.
WRT handhelds/mobile devices - sooner they become totally thin client and anything other than address book and simple email/sms/Internet client is in the cloud, the better.

William PayneJanuary 9, 2014 1:43 PM

A lot of embedded software is written in C or (sometimes) C++, with a smattering of assembler here and there.

What are the best static analysis tools to use to help us make more secure embedded systems?

What specific best-practices can we put into place?

Albert W.January 9, 2014 1:50 PM

@Brian M.
I too am am fan of the ISA550. However, I have been hounded by bugs and have had to disable important functions. But that isn't the worst. Cisco is EOL'ing it Nov'14. Apparently there is no UTM SMB market for Cisco.

DantzlerJanuary 9, 2014 3:04 PM

@hihihi

At the end of one of the videos Bruce linked to, the presenter said that DD-WRT and other open source firmwares were indeed vulnerable. Maybe not to reaver, but DNS rebinding and potentially other attacks.

BlackAngelJanuary 9, 2014 3:32 PM

@BruceSchneier
"For example, one survey of common home routers found that the software components were four to five years older than the device. "

Yes. Mention of one of my favorite drum beats these days.

Router security is **horrible**. Very likely major governments have enabled that process. Home routers are one problem, but so are more major routers. And, suffice it to be said a lot of companies that otherwise may have more modern software will often have patchwork old crappy, effectively home routers at key junctures.

Even when there is no direct vulnerability known for a router, or known, unchanged passwords -- it will tend to be easy to brute force. And the vulnerabilities tend to be wide and many.

Sadly, a lot of people don't "get" the router attack avenue. Even though a large majority of end users have their router on the internet facing attacks everyday.

The scenario is simple:

The late, great R.I.P. - Barnaby Jack way back in the first half of the 2000s showed a great reason why people should care about routers with his exploit code: it compromised the router, overwrote the OS code as exploit, and searched for any file being downloaded downstream which had the Windows executable filetype. If it did, it changed the exe to inject trojan code, thereby infecting every system downstream which used the router.

Spice that up some, put some controls on it, and a remote adversary can trivially get control of a large number of essential systems otherwise hidden behind firewalls.

Or just one key one.

Besides the obvious passive wiretapping, and pinch and jiggle attacks on poorly secured encrypted protocols attacks.

And nm the obvious: you have at any time quite a number of upstream links. And rarely - if ever - will your executable file downloads (and so many types of executables there are, consider browser plugins, to screensavers) be scrutinized for custom malware code no AV company will sound on because it has never been seen in the wild before.


DecadeJanuary 9, 2014 3:52 PM

I am not optimistic about the ISPs, because of AT&T.

Ever since AT&T started doing U-verse, they've been shipping wireless router modems that can update over the Internet. They have that ability. But people keep complaining in forums that certain updates make the routers less stable, and the older models will never support IPv6. Even when they deploy newer routers that support IPv6, they do not activate it in all neighborhoods.

So, I don't trust AT&T.

SimonroJanuary 9, 2014 4:28 PM

I worked in the embedded industry for 17 years, and everything you say about the manufacturer, ODM, OEM chain is correct.

These days even the cheapest flat panel TV has a MIPS processor in it, running some hacked, buggy, old version of Linux.

Higher up the price ladder, they're internet connected, with web cams embedded in the fascia, and ARM chips plus GLES graphics running everything. Same cruddy software underneath though.

The only saving grace is that almost nobody connects them to a network at the moment.

Nick PJanuary 9, 2014 4:40 PM

@ William Payne

There's quite a few. Here's a pretty substantial list of tools by language. Astree, Frama-C, SLAM and Lint-based tools have all proven useful in finding defects.

https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis

D Wheeler has a info packed page on tools supporting software assurance. It's geared mainly toward high assurance so it includes formal methods type tools, too. Quite a few of these are still around and/or usable.

http://www.dwheeler.com/essays/high-assurance-floss.html

Escher Tech has some nice advice on the subject. Lets also not forget NASA's Software Safety Guidebook, a dense but thorough treatment of the topic. The language comparison led me to an even better solution in the next section.

The Truth: C/C++ might be the problem

So, there's your tools. There's also another solution that's even easier: replace the unsafe language with one that's provably safer. That language was specifically designed for real-time, embedded, high integrity, high productivity and maintainability. It reached every goal while being updated regularly with new capabilities. It was used in many projects with most recent being Boeing 777. It's been used in high level enterprise apps all way down to interrupt handlers & OS kernels [1] in embedded systems. A side benefit I can't find my link for was futureproofing: some companies are able to integrate code from decades ago with current language constructs without trouble due to compatibility releases maintain.

Now, I know there's plenty of projects where it absolutely has to be done in C and C++. I also know there's many projects where it absolutely doesn't. I've always said it's easier to write robust software if one uses a language and tools designed to do exactly that. You can also learn it free via GNAT and available tutorials/ebooks online.

[1] A recent example was a monitoring system (see Loseby 2009) built on TI MSP430 processor @ 8Mhz and 2KB RAM using SPARK variant. I recall the Army Secure Operating System being written in Ada on Motorola 68020 processor with 8MB RAM. ASOS was written according to Orange Book A1's very rigorous requirements. So, these are evidence for (a) modern use in a highly constrained device and (b) use in secure low level software.

FigureitoutJanuary 9, 2014 4:51 PM

Yesterday had a local restaurant's payment system down totally (they were taking cash only w/ paper records); in the middle of a payment the system shuts down and started asking for Boot Disk. Guy suspected wifi but he didn't even know the password nor give it out (wonder if smartphone was source attack). Made for a fun day I'm sure.

Dave OldcornJanuary 9, 2014 5:12 PM

An interesting dichotomy is that one of the main causes of a problem here is also one of the things 'defending' the embedded space: the lack of a monoculture.

The existence of thousands of models greatly reduces the value of an individual attack.

In addition, while compromising the middle man has value, it's far less than compromising an endpoint - at least, assuming the inability to obtain signed certificates for a MITM attack on https.

So at the moment, arethe 'higher value' targets in this area likely to be higher-volume product: enterprise routers or (big) ISP-supplied equipment (such as the BT or Sky DSL routers here in the UK, which I imagine outnumber any other router model here by a few orders of magnitude)?

But these entities also have more incentive to develop and distribute patches... so would I feel more secure on one of those, or on my 3rd-party router? I'm not sure I have a good feeling on that one.

AnuraJanuary 9, 2014 6:15 PM

It seems like it all comes down to standardization of hardware, and open source software/firmware/drivers, not just for computers, but for routers, switches, other networking appliances, storage appliances, phones, tablets, video game consoles...

Anything you connect to your network, especially if it communicates to systems outside the network is a potential attack vector. Having closed source software tends to make programmers lazier when it comes to security: "Well, they can't figure out this backdoor, it's compiled in the binaries!" - not that open source magically makes things more secure, but at least it reduces the number of assumptions the developers might make and it means you are not relying solely on the vendor to update the system.

Of course, businesses like having that level of control.

Secret PoliceJanuary 9, 2014 7:29 PM

Another risk is PRNG. These systems are entirely made of flash with no moving parts or operator, and are probably not putting out enough entropy source when switched on and setting up ASLR and other boot randomization every OS utilizes. Somebody needs to put together a kickstarter campaign for an open arm box that runs stock openbsd, then create a management app for it that logs in using passwordless ssh for updating/configuration.


Secret PoliceJanuary 9, 2014 7:31 PM

@Clive Robinson aren't NIC blobs also there to comply with insane FCC regulations? Manufacturers typically have to sabotage their own chips in order to get approval for sale in the US, thus binary blobs are born to prevent anybody being able to use certain frequencies.

MexalyJanuary 9, 2014 9:14 PM

A trend that cannot continue forever will stop. If the parasite kills the host, both die, and some other critter gets the food.

AnuraJanuary 9, 2014 9:45 PM

@Secret Police

This isn't really my area, but one options for these systems could be a chip that contains a small radio transmitter and a lightweight cryptographic psuedo-random function. Hold a state as long as it's powered on, and combine entropy based on atmospheric noise with every call. Not sure how much room there is for additional chips on embedded hardware, though, or how much space it would take.

RobertTJanuary 9, 2014 9:54 PM

Welcome to my recurrent security nightmare, namely someone figures out just how incredibly vulnerable the embedded device market really is.

As others have pointed out your embedded typical device runs some cut down version of *nix, probably never patched since it was first downloaded from who-knows-where, furthermore its poorly understood by at least half the programmers, yet almost all routines are run at superuser privilege level....sandboxing yea right that'll be the day.

Entropy is one of the most difficult things to obtain with any embedded device. Concepts like keyboard latency are meaningless if there's no keyboard and the startup sequence is identical every time. Even when you specifically include real world events in your startup sequence they'll invariably be coded around by some downstream users script, because everyone wants simple plug'n'play systems. Many embedded devices try to generate some startup entropy by including intentional logic race conditions and metastability, PUF's or poor-mans TRNG's. This is where the trouble really starts metastability recovery is not a completely unpredictable event, in most cases it's not even remotely unpredictable, good on-chip TRNG's are not trivial to construct (they cost die area and more importantly they require analog IC design engineering resources), yet they result in zero value for either your direct customer or even for your end customer. Honestly does anyone here know or care about the quality of the TRNG inside their smart TV, wifi-router, smart meter......

Today's embedded device security is a complete joke, trouble is IMHO its actually gotten worse over the last 5 years, this trend will likely continue until for whatever reason the SHTF. This will be the ahh-ha moment when industry outsiders realize just how deep the doo-doo really is.

In case I forgot to mention it the situation just gets worse if you look at telecom "managed devices" they are for more interested in embedding advertising then security.

find: paths must precede expressionJanuary 9, 2014 10:17 PM

In addition to the standard EULAs, routers should come with a written guarantee of X years of firmware support after which they unofficially continue to update firmware for your device.

Solomon PeachyJanuary 9, 2014 11:13 PM

@AndrewHamilton:

No Software is "embedded" in hardware, at least not in a CPU. The hardware always has to boot, start an os and load the software it is to run. Which means that the OS/software can always be updated with patches

This is not true, especially for the IoT paradigm and deeply embedded stuff in general.

I'm working on one project in particular whose software will be running entirely out of Mask ROM. Not flash, because if we put flash on the chip we'd not be able to meet our power or die size budgets. This also means we have to get everything right before we ship, which means some pretty serious testing requirements.

Simply adding the capability of field updateability greatly increases the cost and complexity -- and ironically, introduce a whole new set of remotely-exploitable vulnerabilities.

TorBB UserJanuary 10, 2014 12:10 AM

Forgive ignorance, cause google isn't giving this morning.

What is RNG issue in low entropy systems?

Why not integrate cryptographic RNG into all operating systems? Like OpenBSD [me think?]

CisCOWJanuary 10, 2014 12:49 AM

Remember the WPS vulnerability on Linksys routers, most of them are unpatched and no longer supported but still on sale (retail not refurbished) without any warning.

pianissimoJanuary 10, 2014 2:37 AM

RobertT: The obvious workaround to the entropy problem is to continue generating entropy (from network data, sampled registers or whatever) as long as the device runs, pooling it into non-volatile memory in the chip. Ferroelectric RAM and spintronic RAM are non-volatile and have essentially unlimited write endurance, so they can be updated continually. On startup the device has the same pooled entropy as when it shut down or the power failed.

Nick PJanuary 10, 2014 7:22 AM

@ pianissimo

"RobertT: The obvious workaround to the entropy problem is to continue generating entropy (from network data, sampled registers or whatever) as long as the device runs, pooling it into non-volatile memory in the chip."

That's what I did for entropy in deterministic systems. The start was a seed or even seedfile I generated. This was turned into a pool. The systems operations update a counter that gets cryptomixed with that periodically. The actual numbers come from a CRNG it seeds. The state of this is stored on shutdown. I think I also mixed the time in there to reduce risk of attacks based on improper shutdown.

Another design of mine is to have it remotely pull configuration parameters on boot. One of those parameters is a cryptographic seed. This shifts it to being another system's problem, likely a management node.

65535January 10, 2014 7:52 AM

@ Peter Galbavy

“As we move onto more and more life critical systems either in the home/car/office or attached to the user there will need to be more disclosure much like the calls for opening voting machinery code to inspection. There will have to be a way for independent scrutineers to review this stuff before it becomes dangerous.”

That is a very good question.

WookeyJanuary 10, 2014 8:04 AM

I've been using openWRT on a linksys WRT54G for many years. And it's very good but still doesn't make updates happen automatically. Wider usage of open/DD WRT would certainly help, but doesn't make users get security updates unless they are unusually diligent.

One project that hasn't been mentioned is FreedomBox https://wiki.debian.org/FreedomBox which can be your router/server with really good attention to security and not relying on untrusted services/corporations. It's trying to solve a much bigger problem than just router/embedded security, but it can make a very useful contribution to that too. An ISP deciding that this concept was important/marketable would help get the project moving a little faster and into more people's hands.

Clive RobinsonJanuary 10, 2014 8:12 AM

@ TorBB user,

    What is RNG issue in low entropy systems?

Entropy is in effect "a measure of possabilities" the more of it you have the greater the possabilities available. In Crypto work you have a further requirment on entropy and that is "unpredictability" by other people. There are two ways to achive unpredictability, one is by using the equivalent of a counter and strong crypto algorithm to encrypt it, the second is to measure some physical process that is external to the determanistic processes of the system you use.

The problem with embedd systems are many, firstly they are in this case "mass market" which means thousands of identical systems are sold. Thus unless the manufacturer on "flashing the device" adds unique entropy to each unit, on first customer power up there is no entropy in the system.

Secondly many embedded systems have few if any physical sources to measure and those they do have are usually very low entropy generators, or are visable to others (ie network packets) which breaks the unpredictability rule. In most cases the only unpredictable physical source external to the determanistic system is the Real Time Clock (RTC) if and only if both run concurently (which is not true for many microprocessors, due to the fact they use the same circuitry and just switch the XTAL). However clock drift is incredibly small and of the order of 1 Part Per Million (1PPM) per day...

& @ pianissimo

There is then the issue of "set up time", users are not going to wait for the length of time required to generate around four and a half thousand bits of true entropy, (because arguably the product end of life will arive first). Usually "setup time" includes the generation of PubKeys and AES master keys etc. So there are many many embeded network devices where the PubKeys can with very little knowledge by factored out in the equivalent of seconds...

Whilst the unit could (in theory) carry on generating secure or true entropy and save it to Non Volatile RAM (NVRAM) it's going to be way way to late and only of possible use for "session keys" which as they get encrypted by the PubKey means they might as well be "sent in the clear".

As I said above, the only practical solution is for the manufacturer to "inject" unique entropy into each unit.

The problem with this is you will never know if the method of generation the manufacturer uses is either deliberatly backdoored or the likes of the NSA have the master generator keys from an insider or by black bag job.

Now if anybody produces a new embedded device it's 100% certain the NSA or other Five Eyes intel organisations will get one, and they will pull it apart to find out how it works in very intermate detail. Thus they will know any non unique "magic numbers" or incorectly coded algorithms or protocol weaknesses the device has and thus will at some point as the perceived need arises generate exploit code etc.

One or two of us on this blog have been banging on about these issues for a number of years if you google for TRNG or CSPRNG or "true random" or "entropy" you are likely to find all the relevant blog pages.

Clive RobinsonJanuary 10, 2014 8:29 AM

@ Peter Galbavy, 65535,

    “As we move onto more and more life critical systems either in the home/ car/office or attached to the user there will need to be more disclosure car/office or attached to the user there will need to be more disclosure much like the calls for opening voting machinery code to inspection. There will have to be a way for independent scrutineers to review this stuff before it becomes dangerous.” it becomes dangerous.”

As I indicated in my long post up the top of this page, the only solution is a framework of standards, where the standards do not use technology that is encumbered by patent or other IP system that raises royalties and can thus be used to "lock in" customers and "lock out" competitors.

Clive RobinsonJanuary 10, 2014 9:39 AM

@ Secret Police,

    ... aren't NIC blobs also there to comply with insane FCC regulations? Manufacturers typically have to sabotage their own chips in order...

Yes to a certain extent though the compliance model behind the idea is finaly dying out due to the issues of Software Defined Radio (SDR) which has caused the European and world "spectrum and approvals" bodies to get their knickers in a very tight wad.

It all started around the time of WWI a century ago for "wireless communication" (ie radio) but was based on ideas going back another half century or so with "wired communications" (ie telegraph and phone) and around a century before that with "carried written communications" (ie letters not "printed" newspapers or books).

All Governments have reserved the right to control these communications within their own borders for the purposes of raising revenue and for their own uses. They have also since before then reserved the "right to know" for all persons other than those of Royalty or of Court (ie Diplomats). Due to the shenanigens the Pope and Holy Roman Empire and associated religeous organisations got upto by way of trying to userp the Royal Perogative of Devine Right (what we would now call "national security" issues).

Thus Governments took a dim view of encryption and other security as it was assumed (as now) to be the preserve of "Kings and Scoundrels" (or these days Governments and criminals/terrorists) and those not of Court were assumed to be commiting Treason and thus subject to tourture and agonising and very public execution.

Thus during and after WWI the use of anything other than "plain text" for wireless communications was baned out right. However with the advent of commercial usage it was realised that "privacy of communications" was a requirment. The solution was a typical "Governmental numbnuts" solution, as radio technology was still mainly the preserve of scientists and specialist amatures with the required mix of engineering skills and a "magical touch", the Govs assumed that amatures could be alloted their own spectrum, public broadcasters likewise had speciffic bands as did maratime and commercial intrests. The bands had sufficient gaps between them that "inter band working" with a single receiver was not practical. Thus they licenced receivers to be of certain bands only, and obtaining a receiver without a licence would not be possible.

Sufficit to say that less than twenty years later with the push of technology by WWII the idea was a compleate nonsense.

However due to the "need to snoop" Governments would not counternance "secrecy" so the nonsence of "privacy" by band planning and licencing of receivers still exists today. It became obviouse to all in the 1980's with cordless and mobile phones it was a compleate nonsence when various "rich, famous and news worth" people had their calls intercepted. Thus other means were starting to be investigated (ie weak encryption such as audio spectrum inversion) by Government agencies. But the 90's "digital reveloution" solved the problem --temporarily-- for them, so the nonsence still continues.

You only have to go to some of the SDR sites to get information on USB SDR receivers that cover a large chunk of the bottom 2GHz of the radio spectrum for "pocket money" and Open Source Software to drive them.

Pandora's Box is wide open for radio receivers but still governments are in denial simply because of their paranoid "need to know", the joke of it is that it's WiFi that has the strongest non Governmental encryption, whilst getting a licence to use strong encryption is well nigh impossible for any commercial operation...

It's worth keeping an eye on how governments deal with their "need to snoop" and communications technology because it tells you much of waht they would do with the Internet if they could, and they seem fairly determined to close the lid on Pandora's Box any which way they can because "un regulatedd" (read out of their control) communications scares the political class and those who "fill their pockets" because it give the power of "off message" information moving freely. A re-read of George Orwell's work from the end of WWII will give a good background understanding of politics from this respect.

Robert TJanuary 10, 2014 3:26 PM

@pianissimo

There are plenty of ways to create entropy files but for typical real world systems creating a sufficiently large entropy file is a slow process. The process also requires the addition of some form of Non volatile memory Flash is the typical choice. Now Flash memory has Read/Write cycle limits but they are typically around 1K to 10K cycles so writing the file no more than once per day will fix that limit.

There are some systems that do exactly what you suggest but most dont and the main reason they dont do this is simply that it is not required. There is no spec for a smart-meter (that I'm aware of) that sets a minimum size to the entropy pool. the pool is simply as deep as the chip manufacturer decides is ok, yea that's right the chip guys NOT the system makers typically deliver all the drivers. These days the in the consumer sector name-brands Linksys, Buffalo Dlink whatever usually just add their splash screen to the chip vendors generic product.

The way it works, in the real world, is that Dlink looks for the lowest system price. If a chip vendor has a design that requires FERAM (say $1 extra) than this $1 comes from his pocket. The BOM (bill-of-materials) total cost is whats important. This means the chip maker will make any savings possible. A few years ago 4 layer PCB's were common in consumer electronics, these days they are typically two layer. The reason is simply that the chip guy gets say 50c more if he can use two layer instead of 4 layer.

Take for instance a consumer DSL/Router/WiFi chipset it probably sells for under $2.50 (could be lower I'm not active in this sector) so the package / test costs about $0.50 the chips themselves cost say $1.50, so if all goes well the chip maker will net $0.50.

Now when you come along and say that for just ONE messily dollar we could add FERAM and dramatically improve the system security (a feature the vendor does not even really care about Today) guess how much attention that suggestion gets.....I can assure you it gets immediately filed in the round filing cabinet.

Than of course there is the issue of Patents. WHATEVER you do you will be breaking someones patents so the best solution is to make sure that your solution is the oldest most generic solution imaginable. All innovation will be punished with an ITC exclusion order, BTW the trolls care less about your data security than the chip vendors, so good luck with that suggestion the they surrender patent royalties. The whole Patent issue is part of the reason that chip vendors typically deliver Binary blobs of code to the likes of Dlink. Dlnk maintains plausible deniability and the Trolls have extra work to do before they can prove infringement, this all takes time and during this time the chip guy makes money. Everyone knows they'll eventually get an exclusion order, so the second thing to remember is that all your random number pool solutions need a work-around. this way whatever aspect of your solution that is in violation of the ITC order can be quickly removed. That my friend is cold hard reality in the consumer electronics market place.


HermanJanuary 10, 2014 11:40 PM

"My cheapo NetGear router downloads and installs upgrades."

The problem is that online vendor support isn't necessarily a good thing. If you don't know what is downloaded, then you are completely at their mercy and the NSA or any script kiddie, can compromise the device with a malicious download.

This also applies to auto-updates of Linux, Mac and Windows machines of course. BTW, in military installations those features are turned *OFF*!

I would much rather buy a $300 Linksys router - or better, a much cheaper Soekris router, install my own version of Linux on it and then ensure that it cannot be updated/modified by anyone but me.

Jordan BrownJanuary 10, 2014 11:52 PM

Herman: You hand-inspect every line of code in everything you run?

If so, you've got more time than I do... if not, you're at the mercy of whoever you got the code from.

And, of course, the enormous majority of people in the world couldn't even begin to consider inspecting the code, so they all *must* trust vendors.

GweihirJanuary 11, 2014 12:01 AM

The other severe problem I have often observed, is that the designers of embedded systems are routinely completely clueless about IT security, which leads to insecure architectures, design, crypto, configurations and hard or impossible to patch systems. These people can get things to work, but as to what must be prevented from working (the security POV), they are completely clueless.

Unless that subject is made mandatory for all students of IT subjects, including EE, this is not going to change. Even after that, it will take 10-30 years for things to get better. So while I agree that it is a very high priority to fix this, I do not see things looking up in the next 20 years, and possibly much longer. Personally, I have my own firewall on a Linux PC, as there is just nothing out there that can compete. Even expensive enterprise equipment is looking highly suspicious these days.

What I think is going to happen is a 2-class society: A very small class that actually has reasonably secure IT, and all the rest where the cost of computing slowly, but unstoppable is going through the roof. The second class will include many enterprises.

FigureitoutJanuary 11, 2014 12:55 AM

Gweihir
--Hit the nail on the head...people like my dad who does embedded systems for a living gets angry at me anytime I bring up security, "Get a job" he tells me. He's the kind of person that will wait for something to fail, then fix it; it's just how he is. So I have little doubt my dad's company has been compromised and since we know some sketchy Koreans wanted to buy "just 1" product and then "maybe" buy more; some of these Chinese hackers getting into our systems would easily get in the company.

Then my brother, he should know better since he works for a goddamn defense contractor, emails the unchanged wifi password to our router to the family. Makes me want to scream, but I can't mess w/ the router; they just want it to work...

Bottom line, the "correct implementation" is something of a falsehood b/c no one can say for sure the correct way. Even Bruce himself has admitted he's new to Linux, vulnerable to black bags, and that the computing experience is sheer sh*t. And you better believe it's not being taught in university; they're mostly prepping me to be another code cutter for a big corporation or gov't working on this sausage crap we consider secure systems/engineering...It all sucks and reeks of failure and helplessness.

RandallJanuary 11, 2014 4:00 AM

Bruce,

I think the other angle is to simplify design.

Strip all but the critical features from the embedded device. This lowers the "surface area" available for a security attack.

Some open-source embedded devices are relatively cheap and under 30 US dollars. I'd consider using a write once memory - why allow a hack to rewrite the OS to flash/disk? A board design where OS memory mapping is simplified to allow hardware based hash checking of running processes, similar in theory to TPM, would be a great feature.

NASA used first generation 8080 processors on board the space shuttle not because of feature set but because all bugs were known. The same idea applies to ensuring security on routers and other embedded devices.

HermanJanuary 11, 2014 6:43 AM

Jordan Brown: You are missing the point. If you are really worried about security updates, then you got to download the updates to a secure staging area and *test* it before you deploy it to your equipment.

Automatic updates can be subverted by the NSA/CSE/GCHQ/FSB/RandomScriptKiddies. Leaving auto updates on, leaves you open to a malicious code injection attack and turning updates off and never updating, may actually be better.

DanielJanuary 11, 2014 6:52 AM

For Wi-Fi routers, maybe cheap boards like raspberry pie can come to the rescue?

I guess most people has someone with technicals skills just a few hops away. The boards are readily available in every town (at least here in Sweden).

I think it would be good enough if people who care about security, evern though they don't have technical skill, can easily get a secure auto-updating WI-FI router installed by technical guy/girl in the circle of acquaintances.

name.withheld.for.obvious.reasonsJanuary 11, 2014 7:07 AM

Several years ago as part of my research, I took a position at a power generation plant. For years working in a laboratory developing new systems and systems of systems, it was rare to actually see work from the lab operating in the field. In the two years "on station" so to speak, I had the opportunity as a DCS technician to fully immerse myself in the field of power generation and multiple control system environments. With multiple generation plants I had the chance to understand, in exacting detail (later became a member of an international power working group) the workings of the industry concerning control systems, devices, components, assemblies, discretes, passives, actives, and programmable devices. The experience is worth a formal write-up, for now I just share a few observations.

  1. Private businesses operate the majority of plants country wide, profits drive the decision process
  2. Compartmentalized expertise--from engineers responsible for plant design and operations vary in breadth of experience and skills. This is relevant when a phasor is linked to control system for plant automation. Understanding the operable states of the phasors, transformer and transmission lines, and in service parametric elements comprise a small System of Systems when integrated with a plant and an Independent Service Operator.
  3. Industry players, suppliers of the components and systems deployed in plant environments is a very mixed bag. The difference and emphasis any single vendor can be enumerated sufficiently that performance, security, reliability, and cost can be obvious...many decisions about implementation and purchases is based primarily on cost...not good.
  4. Plant operators see their role as minimizing efforts and costs, thus lawyers will constitute the response to any out of compliance, negligence, or failure to plan or implement robust systems.

  5. The systems in plants tend to resemble a System of Systems, of Systems, of Systems that is not fully comprehended by the responsible parties.
  6. NIST did something different in the draft IR 7628--I was pleasantly surprised. The first draft looked like someone threw-up on a napkin. By their third draft it started to resemble a proper treatment of the topic.
  7. Regulatory environment is considerably insufficient/inefficient and is so convoluted as to be nearly worthless.

  8. For example, WECC in Utah overseas many independent operators in the western United States but really resembles a trade group. The BPA, in Oregon, is the regulatory body that overseas the WECC processes. It's worthless/toothless. Reports about audits and findings tend to say "We found that self auditing small generators represents an area of concern." but does little more. They regulatory trail continues (much like the old Oregon Trail game) with NERC and FERC. Seems that stacking these bodies is just that, a collection of compartmentalized inefficiency.

    A really deep dive regarding security would be well worth my time, the experience was so enlightening from a "here's reality" from the "theoretical" laboratory perspective. The gulf between what is best practices and as practiced, is vast, and wide.

hermanJanuary 11, 2014 8:59 AM

Daniel: People keep missing the point that the auto-update process itself is also a major security hole.

Most/all updates are distributed in plain text and can be subverted. In this case, not updating at all, is better than updating with compromised code.

DanielJanuary 11, 2014 10:19 AM

Herman: So for instance the auto-update in Debian Linux is not secure? I would assume they are encrypted and authenticated with SSL. And "encryption works", according to Snowden.

Do you have examples of auto-updates delivered in plain text? I would be really simple to provide for example an HTTPS interface.

Dave OldcornJanuary 11, 2014 2:36 PM

If the auto-update is done properly - and if the update mechanism can't be compromised.

E.g. by a fraudulently signed certificate - that counts https out (depending on what class of attacker you're defending against...)

tzJanuary 11, 2014 6:09 PM

DD-WRT and OpenWRT don't have any incentive not to update nor to delay patches (beyond validation).

The problem is more "use some proprietary junk on top of opensource", then the proprietary rots.

The liability is broken. One click and a toaster is transmuted into a book (if a toaster burns down your house v.s if you don't like the ending).

Clive RobinsonJanuary 11, 2014 8:00 PM

@ Name.Witheld...,

Whilst your point 1 correctly identified the primary driver of,

    profits drive the decision process

You don't go into the implications of this.

For instance we know that in the very short term cutting all but direct production costs will maxamize profit and thus dividends to share holders which will raise their value and in turn raise renumeration to the directors and senior execs.

However we also know that investing money in upgrades will reduce short term profits in return for long term profits that will far exceed short term profits fairly quickly. And investing in preventative maintanance will reduce significantly the chance of catastrophic failure and the attendant rapid drop in share price and increased cost in litigation costs.

As a --perhaps overly-- general case that companies run by execs without ties to a company will maximise short term cost in any way possible, unlike those with ties such as significant or total ownership who will generaly take the longterm reinvestment view to develop the business.

Thus short term untied execs are without constraint most likely to enter a race for the bottom, timing their departure to be close to peak share price. And like "rats leaving a sinking ship" will leave others to drown.

This raises the question of which sort of constraint would be most effective, direct constraint by legislation on how the business should be run. Or by forcing ties on executives so that the short term race to maximum profit does not benifit them.

Personaly I'm in favour of a bit of both because both have pros when done lightly and cons when done heavily. That is putting in place a lot of legislation means a lot of regulation monitoring and compliance, this builds artificial empires and markets that have to be paid for, as well as providing a straight jacket on developing a business. Likewise forcing ties on executives renumeration via various tricks such as share options makes execs overly defenceive and thus take an overly cautious aproach to business development, as well as making the removal of poor/bad execs more difficult.

Risk adverse execs won't develop a busines and incure lost opportunity costs, whilst risk seaking execs will put the business at significant risk as they remove most if not all of the busines's resiliance.

As was seen with recent hurricanes New York "dodged a bullet" in part due to an earlier heat wave forcing aging infrustructure to be replaced.

RobertTJanuary 11, 2014 11:28 PM

For Embedded systems overall security is only as good as the weakest link, which typically is mutual authentication.

Somehow the embedded device needs to be certain that the update is a valid update coming from an approved source before it even attempts to download or install it. There are dozens of schemes that attempt to solve this problem but they all eventually boil down to both the device and the update system sharing a secret. The easiest solution is to base the secret on some user/internet accessible device ID but it is also the least secure because it is a little too obvious.

Over the years many schemes have been developed from my experience the best use zero-knowledge-protocols and store some secret on the embedded device along with a record stored somewhere (at the manufacturer) that matches device ID to device secret. This is the protocol that I push hardest BUT it is the least appealing method within the consumer electronics space. All they see is lots of ongoing support work (hosting) for which they are never paid anything. So authentication usually boils down to some weak encryption of the device ID nothing more.

name.withheld.for.obvious.reasonsJanuary 12, 2014 12:30 AM

@ Clive
You don't go into the implications of this.

No. Didn't want to be too specific about any one element, could result in the unmasking of my cover. Your observation is correct, checking the exits after squeezing the lemons is a very common strategy here in the states. The implications with respect to critical infrastructure I am sure are obvious to you.

What I want to do is frame the embedded security discussion using a serious real-world application. I will be limiting my disclosure to relevant and helpful items of interest to the less technical crowd without making it sound like Lucy from the Charlie Brown cartoon. Might take me a few cycles to complete but it should be informative.

jayJanuary 13, 2014 12:49 PM

There is, as I understand a similar problem with medical devices. Many of them are based on old versions of windows, etc. and patching is a big problem because you can't do it.

Medical device law is based on historic dedicated mechanisms, changes were simply not permitted unless the change was fully vetted to the FDA. (Applying an OS patch to a device like a CAT scanner is something that you don't want to do lightly).

So what is the solution? Sort of lose-lose.

MattSJanuary 13, 2014 11:16 PM

Bruce,

From a safety perspective (my particular professional concern) as the power and connectivity of embedded systems increases there also start to be very real safety implications for security vulnerabilities.

As an example I believe VP Cheney had the wireless diagnostics on his pacemaker turned off because of security concerns. Would I like a medical device running Linux with a known exploitable security hole inside my chest? Errr not so much...

Unfortunately economics is the cruel mistress here, it's always easy to program in C over assembler so there's a constant gravitation towards 'better' software platforms because it's cheaper and more people can do it, and that in turn drives a race to the bottom.

From the privacy perspective if you conduct a gedanken experiment on exactly how bad it could get if nothing is done you end up very quickly in a panopticon future, of one persuasion or another.

My crystal ball is that you'll continue to see the lines between privacy, security and safety continue to blur especially for embedded systems.

P.S. I do enjoy the cephalopod moments.

Jessica DodsonJanuary 16, 2014 12:36 PM

"we're in for a security disaster as hackers figure out that it's easier to hack routers than computers"

Hack smarter, not harder, right? People are going to look for an exploit any loophole they can find. Why try to storm the walls when you can slip in through the back door?

FigureitoutJanuary 16, 2014 5:22 PM

Fun times ahead...

    Proofpoint, Inc., a leading security-as-a-service provider, has uncovered what may be the first proven Internet of Things (IoT)-based cyberattack involving conventional household "smart" appliances. The global attack campaign involved more than 750,000 malicious email communications coming from more than 100,000 everyday consumer gadgets such as home-networking routers, connected multi-media centers, televisions and at least one refrigerator that had been compromised and used as a platform to launch attacks.

*Emphasis by me*

http://www.marketwatch.com/story/proofpoint-uncovers-internet-of-things-iot-cyberattack-2014-01-16?reflink=MW_news_stmp

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc..