Hacking Critical Infrastructure

A otherwise uninteresting article on Internet threats to public infrastructure contains this paragraph:

At a closed-door briefing, the senators were shown how a power company employee could derail the New York City electrical grid by clicking on an e-mail attachment sent by a hacker, and how an attack during a heat wave could have a cascading impact that would lead to deaths and cost the nation billions of dollars.

Why isn’t the obvious solution to this to take those critical electrical grid computers off the public Internet?

Tags: , , , ,

Posted on March 20, 2012 at 8:52 AM • 66 Comments

Comments

Andrzej Kiesz • March 20, 2012 9:05 AM

Bruce,
Lots of emails have been sent to you about infoShare but seems like you don’t get them. Please check if they went to the spam folder or give alternate email address.

Spaceman Spiff • March 20, 2012 9:06 AM

Besides being totally “security theater”, this would be just absurd, if it wasn’t for the proof that Stuxnet and Duqu have provided that malware CAN infect critical infrastructure control systems… thanks to the NSA/CIA/whoever… What IS the truth here? It is an unlikely scenario that was given to the Senators in this “demonstration”, but not beyond the realm of possibility, unfortunately.

Hyp0tez • March 20, 2012 9:11 AM

Hmm!

Well it is obvious that taking the oper stations of the internet would make it safer. Meerely a few weeks back I was in a controlroom environment where i could access the whole internet without any restrictions.
Thing is, it seems as if the desitionmakers are waiting for something to happen before the right focus arrives 🙂

Royce Williams • March 20, 2012 9:13 AM

Those electrical grid computers are probably not on the public Internet.

What’s more likely is that the same workstation that reach the grid is also used to check email.

One solution would be for the employee to have two work computers: one for working with the grid, and one for working with the Internet. That has its own overhead, but at least it’s a start.

Ben • March 20, 2012 9:22 AM

Yes, the obvious solution is to take the computers used to control the grid off the internet.

This is not being done. Perhaps there is an unobvious reason why not.

E.g. how are you going to patch them? How are you going to reprogram the equipment?

USB sticks and CD-ROMs don’t answer that question – worms travel via sneakernet too.

. • March 20, 2012 9:22 AM

This has the simplest solution that nobody seems to be talking about. All you need to do is remove all liability protection from public utilities and other organizations determined to be “critical infrastructure” in cases where negligence leads to security breaches, such as having internal networks hooked up to remote communications devices or Internet. Done. Once an corporate officer or two are in jail forever and bled of all of their money and property, or the fear of such things happening sets in, things will be just fine.

Edmund in Tokyo • March 20, 2012 9:25 AM

Non-rhetorically, how big is the risk of getting hacked compared to the risk of nobody who knows how to fix the thing being in the office when it breaks?

Royce Williams • March 20, 2012 9:29 AM

Ben, the most common reason for not having two workstations is not being able to convince IT and/or management that the expense is warranted. 🙂

Also, offline patching is a solved problem. For Microsoft machines, WSUS Offline works quite well, using Microsoft’s own published XML patch matrix. Ditto for internal antivirus signature distribution.

Agreed that the other vectors also need to be controlled and monitored – but they are somewhat less casual than clicking on an email.

Matthew Green • March 20, 2012 9:33 AM

Adding liability requirements might help, or it might cause organizations to cover up breaches more vigorously. Probably a little bit of both.

kashmarek • March 20, 2012 9:36 AM

Actually, the reason for the existing situation is that the utility management won’t listen, doesn’t want to hear about the problem (the AOL model), refuses to believe it is a problem, or doesn’t want to have their lies exposed to the public. Most certainly, there is the “another pearl harbor” outlook, which says they can’t make any money fixing this unless there is a disaster, such as 9/11, to get people to fix. We all know where this is headed.

Xebeche • March 20, 2012 9:38 AM

Seems obvious, but even if those systems were off the Internet it wouldn’t stop them from being profoundly vulnerable. The bigger threat is the fact that you can still easily exploit layer 8. Stuxnet came in via USB. Many are insecure by design, many can’t be (or just aren’t) patched, and once inside the boundary – assuming there is one – you often have access to most of the control/operational environment. Dominoes, in no uncertain terms. Simply put, putting them behind firewalls etc. will not protect them from Hanlon’s razor.

GeeTeeEye • March 20, 2012 9:38 AM

Why isn’t the obvious solution to this to take those critical electrical grid computers off the public Internet?
That’s obvious in our ecosystem: If you don’t have access to the environment using the internet, you need another solution – and that will reduce the profit. A glaring violation against the commandment to raise the profit – and possibly something the shareholders expect an explanation for.

GeeTeeEye • March 20, 2012 9:41 AM

@Xebeche: Areed, but disconnecting them from the internet would lower the risk. At least one step in mitigation…

WordNerd • March 20, 2012 9:45 AM

The NSA needs a reason to use it’s $2 billion dollar Utah Data Center. Obviously the only way they can keep critical infrastructure safe is to pro-actively scan every e-mail for viruses and keywords.

If they don’t keep us safe by reading our e-mail, who will?

Matt • March 20, 2012 9:51 AM

I’m surprised the Netbook surge hasn’t translated into a market for cheap Internet terminals (e.g. a Chromebook in an all-in-one form factor)

If you give a utility company the choice between hiring experts to try (try being the key word) to secure their infrastructure versus physically sequestering Internet access from internal network access, they’ll likely opt for the latter… a Chrome terminal without USB ports on every desk would be substantially cheaper than the man-hours it would take to make everything seem bullet-proof.

Note: my use of words like “try” and “seem” is intentional — even experts miss security holes sometimes

Larry • March 20, 2012 9:53 AM

Simply taking those computers off the internet wouldn’t require a multi-million dollar contract with one of the companies that funds CSIS’s “Scare Studies.” James Lewis is bought and paid for by major defense contractors with an interest in legally requiring contractors to “secure” our nation’s infrastructure.

Clive Robinson • March 20, 2012 10:03 AM

Ahh,

1, Public access connectivity.
2, Cascade failures.

Two of my favourite reasons why accountants and shareholder driven directors should not be alowed to play with National Security infrastructures.

Public access connectivity is stupid, it always has been and always will be, it cannot be made secure in any meaningful meaning of the word with commodity systems and that’s the bottom line. Whilst you might show a short term profit due to not having “24×7 on site” engineers you will in the longterm get hosed even if it is just from a DoS attack at a critical time due to say weather or sun spot activity (which is currently predicted to be on the rise).

As for cascade failures they are fairly easy to predict in the general sense but not usually in the specific. In this sense it’s like a chain, you know if you keep increasing the load eventualy one or more links will fail but you generaly don’t know which ones till they do. The engineering solution to this has always been “spare capacity” and “graceful faildown” that can be brought in should problems increase the load above 50-60% unexpectedly. But spare capacity is not “efficient” in short term monetary thinking it’s “under utilisation of assets” which is an anathema to accountants the directors and share holders. However whilst you might show a short term profit due to having “24×7 full capacity” utilisation you will in the longterm get hosed even if it is just from a simple fuse blowing and throwing more than 100% load onto other parts of the network, especialy at a critical time due to say weather or sun spot activity (which is currently predicted to be on the rise).

So lets look what might happen without crackers or others attacking the National Infrastructure,

Sun spot activity, reduces communications capacity on the Internet. ISP’s and backbone operators shunt load to any available capacity. Thus the responsivness of communications drops and due to retries gets worse not linearly but according to a power law (effectivly exponentialy).

The lack of comms in say the power grid gives rise to fall back onto the Internet etc which means control function becomes almost impossible. This prevents timely switching/shutdown of parts of the power network. And due to being run at full capacity a minor spike due to the same sun spot activity causes a trip out that gets thrown onto a major powerline etc that melts out etc (this has happened a number of times have a look at Newzeland and Canada for examples). The rest of the network cascades and also trips/melts out. Whilst a trip out can be reset melts cannot, and due to “startup inrush” the chances are the network cannot be brought up untill lots of equipment has been manually turned off.

Now have a look at what happens to the directors and shareholders of these companies that have had nice short term proffits for a few years whilst the network is mismanaged to high utilisation and significantly increased risk of major failure. They melt away into the night because they don’t have the reserves or resources to repair the netowrk and the lawyers have effectivly put them and their profits beyond reclaim. What happens well you the consumer have been “turned over” and left “high and dry” with only the “insurer of last resort” between you and pre-victorian living standards. But… due to various financial crises in recent times the cupboard is also empty at the insurer of last resort or (as in the case of Greece) defaulting on it’s loans.

What happens next? well you’ld better hope the Chinese with their several trillion dollar cash reserves mountain are prepared to give you “easy terms”…

Which surprisingly I suspect they probably will but there will be strings, such that the US dollar ceases to be the chosen trading currency…

Carrie Butler • March 20, 2012 10:15 AM

The Chairman of the Senate Intelligence Committee de-prioritized the threat and explained his reasoning. If the employee clicked on the email and the electrical grid went down, it would have taken the internet down with it. The internet going down could happen before the click action on the email fully finished completion, thus negating the nefarious plan. And if this reasoning is incorrect and the internet went down, the hackers would not be able to send out other emails to take down other grids (because the internet is down!) and the problem would be self-limiting. I miss Strom Thurmond.

Julien Couvreur • March 20, 2012 10:36 AM

I am sure there are lots of vulnerabilities, both in government and private infrastructure. But fixing them has a cost. For this reason I don’t trust a politician’s assessment of possible trade-offs, as they do not bear those costs.

Fixing incentives by aligning responsibility and accountability is often the best available meta-solution. It won’t be perfect, but the forces at play should push things in the right direction with reasonable relative priorities. Large and cheap problems are low-hanging fruits for owners to improve their product (unplugging the critical machine from the Internet).

As pointed above, special liability protection is a red flag. Another similar situation is apparent with nuclear power (which externalizes its insurance on government). Same thing for banks btw…

Been there • March 20, 2012 10:56 AM

I used to work for a utility and recommended exactly this when they changed scada to tcpip. Fell on deaf ears, all the way to the CIO.

KH • March 20, 2012 11:07 AM

Bruce,

a) “we don’t have time for rational solutions!” (quote stolen from G. Carlin)

b) who would then get the multi-billion dollar contracts on securing the infrastructure (and thus create economic stimulus)?

c) by what threat would then administrative budgets be inflated?

Brandioch Conner • March 20, 2012 11:16 AM

Again with they hype.
But hype sells product.
And sales mean profits.

If it is possible to do that then those systems need to be completely isolated from the other systems. Completely isolated.

Put the computers and switches and such in metal boxes with locks on them if that is what it takes.

You want a SINGLE route for any changes and you want that route to be heavily monitored and require multiple signatures to get access to.

Workstations are cheap. Done correctly (securely) they are even cheaper.

The only reason not to secure those systems is because Upper Management knows that they will not be held personally responsible for any incidents.

TS • March 20, 2012 11:21 AM

@Matt

“a Chrome terminal without USB ports on every desk would be substantially cheaper than the man-hours it would take to make everything seem bullet-proof.”

No, it wouldn’t. If you consider just the hardware, sure. But it’s never just the hardware. There’s a cost just to add the items into your inventory. Support costs. Management costs. Maintenance costs. Not to mention the man-hours put into securing the new system… because there’s no such thing as a secure system right out of the box.

Dylan • March 20, 2012 11:38 AM

Simple answer: Because it would be inconvenient (and probably costly) to the people that maintain and use those computers. And a real inconvenience today is considered a bigger problem than a possible hacker attack in the future. Especially since there is no guarantee the hacker attack is going to actually happen.

obviator • March 20, 2012 11:42 AM

Bruce, while being off the public internet would help, it is not a complete solution. Just ask the Iranians with the fried centrifuges.

anon_for_reasons • March 20, 2012 12:35 PM

The solution is obvious, but it fixes the wrong problem. If you don’t want the systems hacked, take them off the Internet. Or at least off the public Internet. Or put a firewall on them to start with. Hacking is a problem, but it’s a symptom of a larger problem…

As someone that used to work in SCADA until relatively recently (but not the power side) I have many memories of authoring and enabling control systems over the Internet. Without encryption. Over plain old tcp/udp, and often HTTP. At one point we had an issue with an ISP’s web proxy caching a poorly written control page.

Some of them could be controlled by text message.

I saw one that did its report and control over plain old FTP — sending reports and fetching configuration. That at least had some sort of authentication.

Some clients used VPNs for connectivity. PPTP over unpatched windows XP–no firewall, of course. Most just didn’t care and would plug directly into a cable modem to save on communications cost. Others would use wide open wireless with repeaters.

You’re trying to solve a security problem, but the real cause of the issue is at OSI layer 8 (management) or possibly layer 9 (law). Quite simply, it was more cost effective.

So the question is — is security too expensive? Is it not easy and commodity enough to integrate into base systems? If it is simple enough to include, is the processes and time those processes require cheap enough? Quite simply — the 15 minutes it might take to email a password and file it in corporate is not worth it to management.

I was paid:
– Not to develop secure systems. Security was removed from the specification entirely.
– To take / guide others in pulling security out of the contracts and add ‘best effort’ and ‘no fault’ clauses. We won on cost almost every time.
– Not to implement even rudimentary VPN level security (that would have taken the field tech. time and training in an extra debugging tool)

Management at the clients successfully externalized nearly all security incident costs. If they’re breached — they don’t pay for an investigation, the taxpayer and FBI does. They don’t have to talk about it or disclose it — it’s not customer data, and it’s a matter of national security. As the developer, I wasn’t at fault — I’ve got the email saying not to even use passwords on my control system (you better believe I saved that email to a thumb drive and took it home for safe keeping). And the salesman has the contract specifying only connectivity and actual cost damages.

The fiscal cost of a breach is a possibly bad quarterly report with no disclosures, and a few hundred hours of overtime for some field crews to flash firmware and replace drives.

The cost of security — feature development time, new training, firmware upgrades, distributing passwords to every secretary to assist the field crews, interoperability and testing, integrating with existing systems…

Without regard to whether I was actually technically capable of fixing such systems — the fact is, I was paid to spend effort elsewhere.

Johnston • March 20, 2012 1:13 PM

Engineers and technologists need to assert themselves and stop letting salespeople and lowest common denominators set policy. This would alleviate lots of problems.

The story related by anon_for_reasons at March 20, 2012 12:35 PM is going to bite us hard in the future.

foo • March 20, 2012 2:42 PM

Re: Being paid not to include security because of some sales weenie’s oily promises … Marketing is the 5th Level of Hell. I truly believe that. (And I work at a marketing company.)

Brandioch Conner • March 20, 2012 2:46 PM

@Dylan
“And a real inconvenience today is considered a bigger problem than a possible hacker attack in the future.”

Exactly.
The hype is to sell products.
The products provide “plausible deniability” in the event of a successful crack.

The laws for liability in these cases need to change before we see any improvements in security.

It’s a case of “compliance” being the end goal rather than a measure of your security model.

wonk • March 20, 2012 3:01 PM

Bruce,

There are people in the electric power industry, specifically working to secure those systems. Not the consultants, not the managers, not the wannabes, not the academics, and not the politicians, but the few in each company that know the whole story. They hardly ever attend conferences or deliver presentations but are extremely knowledgeable.

There are things they worry about, but it’s almost never the things in the news, on the blogs, or in the movies.

While none will talk with you under the auspices of their organizations, maybe you could find a way to have lunch with a few of them. You could be their voice, going beyond fear, knowing the secrets and lies, and clearly showing the liars and outliers.

Clive Robinson • March 20, 2012 3:01 PM

@ Johnston,

Engineers and technologists need to assert themselves and stop letting salespeople and lowest common denominators set policy

What a laudable idea, just one problem how do you think they are going to keep their jobs?

The simple fact is the law requires the engineer to do the director or their apointed representatives buiding and the law requires the director to service the shareholders in the best way possible.

Now ring of iron or no ring of iron the only time an engineer is aloud to baulk at such orders is if he can show that it is dangerous in a way that a judge will accept. Anything less than that then he’s in a whole world of hurt and you can be sure that he will be made to feel that hurt over and over again such is the power of the shareholder to get maximum return for minimum effort.

bob • March 20, 2012 3:30 PM

If only there were some sort of wires connecting all the powerplants on the electrical grid to each other, that could have information superimposed on them (like my 1982 cordless phone did) so the plants did not need to hook up to the internet…

GarciaFan • March 20, 2012 3:45 PM

It’s not that simple. I happen to work at a utility and we do use separate workstations and networks for general business use and control. There is no way we could air gap our control systems completely due to the way the business works these days. It used to be the case, but our government has determined that markets are better than a bunch of old school regulated utilities. Since the energy has to be bought and sold in a market a company would be at a huge competitive disadvantage if it couldn’t use that data as pricing changes every 5 minutes. An example of current energy prices can be seen here.

RH • March 20, 2012 3:53 PM

People are commenting that, because viruses like Stuxnet have proven that an airgap is not a complete layer of protect, providing an airgap is nothing but security theater.

I disagree. Just because its been proven once that a determined spy can install a camera in your bedroom does not imply that its an acceptable plan to simply do your fornication out in the open.

Michael Toecker • March 20, 2012 4:00 PM

More than likely the email was used to take control of the engineer’s work computer, the one used for email, web, etc. Then, it waited for him to either remotely connect to his control system, or to bring a USB/CD into the control system network.

It was also, most likely, a paper exercise since a cyber weapon with the smarts to sit in wait for a specific condition (like crashing a control system) isn’t going to be trotted out in front of god, congress, and everybody.

The control system is typically located within a network not directly connected to Internet, but is accessible through certain paths by Internet capable computers. In other words, it’s basically a secondary infection, but intended to be a primary attack path. These paths are often via remote access (a la RDP) or via CD/USB.

Getting these systems off the Internet isn’t the magic bulllet, though it is a required first step. Stuxnet proved that, the best theory is that the Iranians were infected by programs that were brought into the control system, not through direct access. That was why it was so sophisticated, it had to do everything right without being able to call for help.

It’s just more complicated than getting things off the Internet. Remember that.

Michael Toecker

David • March 20, 2012 4:03 PM

“One solution would be for the employee to have two work computers: one for working with the grid, and one for working with the Internet. That has its own overhead, but at least it’s a start.”

Virtualize them, and there’s less security but the overhead becomes minimal.

David • March 20, 2012 4:04 PM

“Once an corporate officer or two are in jail forever and bled of all of their money and property, or the fear of such things happening sets in, our infrastructure will collapse because no one will be willing to work on it.”

FTFY

CRossi • March 20, 2012 4:22 PM

I’m with RH on this one. Reducing attack surface should be the focus. As vulnerable/old/unsupported as many of these SCADA systems are, limiting access to the internet should be a clear first step. If some systems need access to the internet, then providing that access, limited down to exactly what access is required, would seem reasonable. Unfortunately, too often it seems like the real reason these systems are exposed to the internet (either directly or via another system) is provided is because users need access to the internet, not the systems.

. • March 20, 2012 4:36 PM

GarciaFan:

Please clarify “there is no way we could air gap our control systems completely”.

For everyone else who says that air-gapping is insufficient because of things like Stuxnet, a whole lot of historical success disagrees. Security policy works wonders when everyone agrees and it comes from the top. Poor practices are more infectious than an Internet worm, and when the C-level suits cut corners, everyone else will. No security best practice will beat out an angry boss and a fearful slave, er, employee. Corporate policy and financial laws need to put the needs of the many above the needs of the investors for critical infrastructure. Consider it a form of semi-nationalization, perhaps. It doesn’t matter. People die when the power goes out, and the power only goes out because the money went to profit instead of keeping the lights on, and that cascades into everything. It’s as simple as that. Once profit trumps propriety, you have what we have today. Which actually critical utility hasn’t posted a profit, given dividends, or otherwise defrauded their customers and/or investors via devaluation, acquisition, or corporate compensation, in the past ten years?

Joel • March 20, 2012 4:36 PM

A bigger question is:

“how much are we willing to pay for electricity?” (as in, all the time).

Clive Robinson • March 20, 2012 6:50 PM

Folks when talking about “air gabs” you need to remember it’s not just the Internet you need to “air gap” from, it’s any and all publicaly accessible networks.

And by “public” we realy mean anyone and everyone and thing that has no proper reason to be on the network corncerned.

Even VPN’s are not a good idea as they travel the same wire that’s publicaly accessible, and in some places so do the phones and all sorts of other sysstems.

Today we are even talking “power lines” as well as “private” radio networks.

However the good news is even though air gaping via a whole host of not easily recognisable networks is possible it’s very far from easy. It usually requires some things to happen,

Firstly some kind of bridge (hardware or software) be it wittingly or unwittingly installed by an insider or person with inside access even briefly (don’t forget the cleaner and security guards can be more tech savey than you think and can plug in a WiFi AP so they can get accesss to their smart phone in the break room).

Secondly an attacker would generaly need very detailed information about the system, and if configured correctly you can make “on the wire enumeration” quite difficult.

Thirdly a not very common skill set.

Sadly though there is still lots of bad news…

In the UK the major utilities (gas, water, electric, telcomm and transport) use VHF/UHF and low end microwave radio systems. Whilst it might be illegal to listen to them they are easily identified with quite eaisly aquired equipment. Such as a Radio Data Test Set or 0-3Ghz spectrum analyser with baseband output to audio input to a PC running “software driven radio” or “audio analyser” software or at a pinch one of half a dozen easily available radio scanners. You don’t even have to guess very hard at the frequencies in use because “band planning” information is often publicaly available at some level, and simple eyeball observation of antennas and other equipment tells a lot. Sometimes “makers names” are visable and a quick hunt in related trade information elicits a lot of information as does a simple phone call to their technical sales people.

I know from experiance there is a hill not far from where I live that you can easily pick up most of the various radio networks in use. Most use very very standard protocols. Some use mods of AX25 or IP or simple RS232 over FSK or standard 2400baud or less Bell/CCITT modems. Likewise the data on top tends to be in either a very standard form or human readable, usualy with zero error corection let alone security…

More interestingly if you chat to people in the trade you find out that the equipment gets stolen or damaged on a frequent basis thus ends up in all sorts of places for sale.

Quite often these radio systems are “directly connected” to plant equipment unlike SCADA systems managment ports thus they are actually easier for an attacker to deal with as they are often single or very few actuators or sensors involved with each link…

Securing all of this is a major undertaking and will even if it were mandated today take ten to twenty years to secure at all levels, and if you leave jusst one level unsecure but accessable in some way that’s where an attacker is going to stick the tip of their lever.

martino • March 20, 2012 8:05 PM

Uh, just buy power from Canada – oh wait, you don’t have to, we’re literally paying you guys to take it because we totally overshot the runway (I think?). Only a reactive solution but I gotta get Canada in a post here and there… ;o)

e • March 20, 2012 10:59 PM

What would be the time frame for a bug to cause damage?
You could network and open everything up with a centralized place that pings and grabs settings, if any are wrong, just update with correct settings.

If the timing is quick enough you just need to lock down one central point, and minimize dos attacks

wkwillis • March 21, 2012 1:10 AM

I’m a security guard. We are 90% there for security theatre, 9% for spotting fires and leaks quickly before the cost mounts up, and about 1% for keeping the homeless out of the dumpsters so they don’t hurt themselves and sue us.
We had a guy show up at our site in our uniform, but he wasn’t one of us.
What a clown.
A good business suit and you can tailgate inside most of our sites. Why bother to make up and sew on a shoulder patch?
Sneakernet allows an attack, airgap or no airgap.

Simon • March 21, 2012 6:56 AM

Before the internet we got along just fine with offline utilities and government departments.

I visited an offline government agency in Nepal not long ago – no money for internet or email. I was struck by how much work they could do without idiots pestering them all day.

Joakim • March 21, 2012 7:49 AM

A bigger question is:

“how much are we willing to pay for electricity?” (as in, all the time).

No. the question is:
“How much are we willing to pay to minimize the risk of not having electricity?”

Bounce • March 21, 2012 9:34 AM

Yes, the obvious solution is to take the computers used to control the grid off the internet.

This is not being done. Perhaps there is an unobvious reason why not.<

Speaking as someone working in “The Fed” I can easily see that the “unobvious reason” is that there’s push back by end users on such efforts. IT attempts to stand its ground but management’s feet-of-clay fail to provide the needed support. Eventually IT staff get tired of being slapped around and stop trying to enforce policies designed to prevent patently obvious exploits.

Until dollars or lives are lost, IT ends up yelling into the wind.

Doesitmatter • March 21, 2012 10:29 AM

“The answer is pretty simple. People need constant access to Facebook.”

^This.

The grid can’t function without facebook.

fly • March 21, 2012 1:43 PM

“At a closed-door briefing, the senators were shown”

“The very word ‘secrecy’ is repugnant in a free and open society; and we are as a people inherently and historically opposed to secret societies, to secret oaths, and to secret proceedings.”

  • John F. Kennedy

Varjohaltia • March 21, 2012 6:10 PM

Here’s one aspect. At my day job, several infrastructure services for our large campus have been outsourced. These outside companies, often using a cable modem provider or similar for their offices, then need to have holes poked through firewalls so they can get to this equipment… which oh-so-often has rotten security to begin with (such as: must run on this version of XP without firewall or AV, and only supports one account and password).

Brandioch Conner • March 21, 2012 10:30 PM

@Varjohaltia
Yeah. Been there. Done that.
It’s way too much money to put in another ISP connection for vendor equipment.

I’ve had one manager say that there wasn’t a problem because “their equipment only connects out”.

Computer security is easy.
But skipping it or doing it wrong is even easier.
And it can be less expensive (until you’re cracked).

Andy • March 22, 2012 7:28 AM

I’m more worried about repeating the Sarbanes-Oxley debacle in infrastructure security. I’m already battling non-technical functionaries trying to check off boxes on security spreadsheets. In my case, they are actively fighting those who know what is actually needed and are being prevented from implementing it.

Paul Haas • March 22, 2012 11:39 AM

Utility control rooms need real time access to weather information, earthquake, and tsunami information to know what actions to take during emergencies. When things go wrong, they need to be able to communicate with the emergency responders.

They shouldn’t have to use the same computers for that, that they use to control the plant.

, • March 22, 2012 4:08 PM

“The powers that be set him straight, as I recall.”

His words and others like his are still with us.

Michael Lynn • March 22, 2012 4:10 PM

Having worked in that space doing security (or trying to) I can tell you the most common complaint is cost. They have lots of assets that they would like to centrally manage. At the same time their rates are set for them under regulations so they’re all very cost conscious. The problem is that you talk to them about potential risks and they weigh it against the costs and the costs seem to almost always win.

Part of the problem is that people (often of, shall we say, management age) tend to view the threat from computers/networks etc in this strange dual and conflicting way. On the one hand they view the threat to the systems they understand as being way over stated (even when they are not). And when dealing with threats in a space they don’t personally understand well they way over state the risks.

Jonathan Wilson • March 25, 2012 7:30 PM

I agree with other commenters that the simple solution to “infrastructure security” is to make utility companies liable if their systems fail (power outage etc) due to the lack of security.

That will force the utility companies to spend the money on security they are currently unwilling to spend.

i.e. you make it so that not spending money on security becomes more expensive than spending money on security.

Andy • March 26, 2012 4:28 AM

As far as I remember the SCADA system I worked on was not in any way connected to the Internet. The facilities where the servers and the clients were, was EMF secured, and patches were applied only after rigorous tests on different machines. For programming and configuration You had to physically walk to the rooms were the clients were. sigh The old knowledge seems to disappear faster and faster. Of course shall a SCADA system of that magnitude never ever be in contact with the Internet, in any way.

Also_Anon_For_A_Reason • March 28, 2012 9:59 AM

I currently work in SCADA comms for a utility, and I can confidently agree that the reason for infrastructure insecurity lies in OSI 8 (Management). Specifically, I can put up a SCADA site on public switched internet for $20/month, but construction of a new tower to reach the area with appropriate power and connectivity to my existing private microwave easily exceeds $500k.

I have made the argument that we should keep everything possible away from the public internet, and it has fallen on deaf ears. It’s all about money, and about executives affording trips to Cabo while working pukes can’t afford the time off to get dental work done, much less the fiscal cost of same.

John Nagle, Silicon Valley, CA • March 29, 2012 11:33 PM

Most of the comments here don’t reflect much knowledge about how the electrical utility industry in the US works today. Electricity is a commodity produced by multiple producers and bought by multiple distribution utilities. There are markets, public market data, and considerable transparency of grid operating information. That’s all on the Internet, as is the bidding process.

Here’s the current, live operating data for the power grid for the Eastern US. This is an elaborate dashboard, in Flash, that shows much of what the people in the dispatching center see. (It’s not for the naive user. You can take the online course PJM 101 if you’re really interested.)

Normal operation involves extensive data transfer to hundreds of parties over the public Internet. All this is done using XML and SOAP over HTTP under SSL using basic realm authentication. Incoming data goes through a format checker, then a business rule checker. Any problems cause a total reject of the data. The formats are public. There’s little “security by obscurity” here.

If there’s serious trouble, the control center can order “non-economic operation” and issue direct orders to generating plants. Worst case, they can order “conservative operation”, which means every generating facility comes on line to at least idle level, or gets ready to do so quickly, whether it’s needed or not. This can be done with minimal central coordination. They have a dumb “all-call” voice system that links all the major players, backed up by Iridium phones.

PJM has their computer security examined by at least one outside group.

So that’s a bit of how it really works.

Mark Twain • April 15, 2012 4:39 PM

Didn’t the United States fix those weaknesses after 9/11. Honestly, I don’t know very much, but I’d like to assume that it’s not as easy as accidentally clicking an e-mail.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.