Schneier on Security
A blog covering security and security technology.
« Combined Taser and MP3 Player |
| Hacking Thermostats »
January 22, 2008
Hacking Power Networks
The CIA unleashed a big one at a SANS conference:
On Wednesday, in New Orleans, US Central Intelligence Agency senior analyst Tom Donahue told a gathering of 300 US, UK, Swedish, and Dutch government officials and engineers and security managers from electric, water, oil & gas and other critical industry asset owners from all across North America, that "We have information, from multiple regions outside the United States, of cyber intrusions into utilities, followed by extortion demands. We suspect, but cannot confirm, that some of these attackers had the benefit of inside knowledge. We have information that cyber attacks have been used to disrupt power equipment in several regions outside the United States. In at least one case, the disruption caused a power outage affecting multiple cities. We do not know who executed these attacks or why, but all involved intrusions through the Internet."
According to Mr. Donahue, the CIA actively and thoroughly considered the benefits and risks of making this information public, and came down on the side of disclosure.
I'll bet. There's nothing like an vague unsubstantiated rumor to forestall reasoned discussion. But, of course, everyone is writing about it anyway.
SANS's Alan Paller is happy to add details:
In the past two years, hackers have in fact successfully penetrated and extorted multiple utility companies that use SCADA systems, says Alan Paller, director of the SANS Institute, an organization that hosts a crisis center for hacked companies. "Hundreds of millions of dollars have been extorted, and possibly more. It's difficult to know, because they pay to keep it a secret," Paller says. "This kind of extortion is the biggest untold story of the cybercrime industry."
And to up the fear factor:
The prospect of cyberattacks crippling multicity regions appears to have prompted the government to make this information public. The issue "went from 'we should be concerned about to this' to 'this is something we should fix now,' " said Paller. "That's why, I think, the government decided to disclose this."
An attendee of the meeting said that the attack was not well-known through the industry and came as a surprise to many there. Said the person who asked to remain anonymous, "There were apparently a couple of incidents where extortionists cut off power to several cities using some sort of attack on the power grid, and it does not appear to be a physical attack."
And more hyperbole from someone in the industry:
Over the past year to 18 months, there has been "a huge increase in focused attacks on our national infrastructure networks, . . . and they have been coming from outside the United States," said Ralph Logan, principal of the Logan Group, a cybersecurity firm.
It is difficult to track the sources of such attacks, because they are usually made by people who have disguised themselves by worming into three or four other computer networks, Logan said. He said he thinks the attacks were launched from computers belonging to foreign governments or militaries, not terrorist groups."
I'm more than a bit skeptical here. To be sure -- fake staged attacks aside -- there are serious risks to SCADA systems (Ganesh Devarajan gave a talk at DefCon this year about some potential attack vectors), although at this point I think they're more a future threat than present danger. But this CIA tidbit tells us nothing about how the attacks happened. Were they against SCADA systems? Were they against general-purpose computer, maybe Windows machines? Insiders may have been involved, so was this a computer security vulnerability at all? We have no idea.
Cyber-extortion is certainly on the rise; we see it at Counterpane. Primarily it's against fringe industries -- online gambling, online gaming, online porn -- operating offshore in countries like Bermuda and the Cayman Islands. It is going mainstream, but this is the first I've heard of it targeting power companies. Certainly possible, but is that part of the CIA rumor or was it tacked on afterwards?
And here's list of power outages. Which ones were hacker caused? Some details would be nice.
I'd like a little bit more information before I start panicking.
EDITED TO ADD (1/23): Slashdot thread.
Posted on January 22, 2008 at 2:24 PM
• 50 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
"I'd like a little bit more information before I start panicking."
Uh... you seem to be missing the point. The government wants us to Panic... PANIC, I TELL YOU!!!
Oops... I think I just Sith'd my pants...
The real concern is that the scada networks are migrating to TCP/IP, which presents an exponential increase in attack vectors.
Is the real issue SCADA vulnerability? Everyone who has examined the issues knows that SCADA is wildly vulnerable.
As part of a pentest at a nuclear plant my own company demonstrated how one could physically enter a neighborhood relay station undetected, tap into the SCADA network using default passwords documented in manuals left on site, and obtain central SCADA passwords using a trivial, long-documented, easily-patched-but-unpatched vulnerability from a SCADA vendor platform, and crack them in 0 time.
After being told repeatedly at a client that the SCADA network was "completely separate" from the corporate intranet I asked the chief network administrator why his Windows NT 4.0 desktop had two network cards. He told me one went to the SCADA network, and one went to the intranet. When I responded that I thought the networks were completely separate he sneered at me and said, "I have a ROUTING TABLE."
So it's not news that these networks are vulnerable. The news is that they're STILL vulnerable.
MUST we have a major incident in order to undertake basic reforms?
According to Bill Hancock, we'll need THREE major incidents: the first is an anomaly, the second a coincidence, and only the third will indicate a trend.
You'd like more WHAT? Only terrorists need to know the details of how such things (might) work.
> crack them in 0 time.
Wow. You guys sure have fast sneaky intruders / user guide locators / manual readers / patch detectors / hackers.
I could only take over your relay station in 3 time.
If someone suffers an attack and there's no disclosure, Super Schneier to the complaint rescue! If someone suffers an attack and there is disclosure, Super Schneier to the complaint rescue! Super Schneier needs no rest! Super Schneier, fly, write, fly, Super Schneier!
Sorry. The image of Bruce in a cape was too vivid to keep to myself.
Has a comment from Michael Tanji, who's worked with the speaker at SANS in the past.
Quoting: "Having worked with Tom Donahue on these and related issues in the past, I regret to inform conspiracy theorists that he is virulently allergic to hyperbole. That he might be making these statements lightly are about as likely as any sane person playing Russian roulette with a semi-auto pistol. I’ve long been a skeptic of claims about being able to shut down the world from the ‘Net, since doctored exercises are not real life (among other reasons), but after today I’m starting to come around to the idea that the ignorance or intransigence of utility system owners just might merit a more robust response than has been undertaken to date."
I wonder how they untraceably paid out the ransom...
The number of places I have seen where people have plugged in various SCADA equipment blindly into a raw network makes me believe that there is some truth to it. My favorite was the guy who plugged in the windows box that controlled various items into the network so that he could watch his movies from the network. When the firewall wouldn't let him do that, he installed some botware to bypass it.
Or the person who installed MyPC on a bridged box so he could control it from home.
Or the guys who blindly plug things into a network because they see a network port.
The worst thing was when managers quote Bruce out of context on why they shouldn't worry about it.
It does help sell tickets to SANS conferences and training.
The thought that SCADA systems would have network vulnerabilities is unthinkable (perhaps even laughable). These are not public systems and do not need to be so.
Air gap - The simplest security solution is usually the best.
Looks like security theater. Apparently, someone has a new system they want to sell, and this is just the ticket to scare everyone into buying. We'll probably find a back door in that system, just for the the aforementioned security agency.
"I'd like a little bit more information before I start panicking."
Yes, yes. I like to know where the shots are coming form before running with my head cut off.
Bruce ... did you consider the possibility that they were really viewing the other Bruce's recent flick :-)
In half of the 3rd world, power is so unreliable that if you blew up all the power plants even their governments won't know for months.
I want to know why this is surprising. Everyone wrestles with the balance between wanting the convenience of being able to access their work from everywhere on the planet, with the risk of their work server being on line, and thus vulnerable to attack. I'm guessing they're coming out with this because some of the power companies aren't going to take the necessary security measures until something really bad happens. I mean, that would be a pain! They might have to actually go into the office to fix things, instead of doing it from home. And it would be expensive! They'd have to license security software, and maybe hire an actual Computer Security Dude. Where's the profit in that? Oh...right...
It's risk versus reward. Right now, they think the risk isn't high enough to spend the money to fix it. From a professional standpoint, you should be trying to convince the power companies that they *do* need to hire a Computer Security Dude. I know of a couple of them... :-)
Bermuda does not allow online gaming or porn sites to be hosted on island. Bit of fact checking required here methinks.
"I think they're more a future threat than present danger"
Shouldn't we start worrying about future threats now other than start worrying about them when they are imminent dangers?
Panicking? nope, not really.
I think this is like an inverse security theather. sometimes you need to worry about an inexistente threath to be able to secure your self better.
And scarrying the power and water companies a bit in order to get them a bit more secure (sometimes, just making them consider the fact that they are not 100% secure, is already good enougth) is not the same thing as scarrying citizens in orther to rob them of their rights.
Just my opinion
Glad to finally see a security expert of Bruce's stature take on that FUD spreading moron Paller from SANS. He's been going around for quite awhile now with this "chicken little" act in a lame attempt to hype the importance of his personal SANS agenda. The CIA guy would have been better off keeping quite if he's not going to release the details of these reported attacks so that those of us in the SCADA security field could then use the detailed info to better protect our systems.
There are lots of us in the SCADA security field who have a very good understanding of the real risks to our systems and we have taken prudent measures to minimize those risks. What we don't need is outside sensationalists creating cyber security mountains out of molehills.
@PlannedBrownout: The scenario you described is still extortion against utilities, it's just a much slicker variety of extortion carried out jointly by the CIA, and the folks promising to fix the problem (not that I'd suggest that such a thing would ever happen. We can trust the CIA because they're part of a democratically elected government -- do sarcasm tags work in these comments?).
Suppose such a situation does exist and some power companies are being held to ransom for "Hundreds of millions of dollars".
Spending a fraction of that amount to hunt down, eliminate the culprits and recover the money would seem to be a good investment.
Spending an even smaller amount to improve security would be an even better investment.
How to secure control devices on a budget.
Take a WRT54GL or other cheap OpenWRT compatible router device. (Disable wireless.) Install OpenVPN. Put it between the "plaintext" device and the public Net. Configure the VPN. Run a pentest, to be sure. Done.
There are better solutions. But this one looks like a decent low-cost stopgap.
Of course it can be DDoSed rather easily. For that case, if the situation requires it, you may like to add a cellphone module, with appropriately secured access (again, e.g. by leveraging OpenVPN) as a backup access line.
For even higher reliability, use a dedicated cabling. Also, design the device for safe behavior in case the comm line gets cut, to degrade gracefully in case of comm loss.
Legalize marijuana, now there's a power hack! :P
What a load of rubbish! More idiotic paranoia from the home of the 'brave' with the added advantage of having some 'external' enemies to blame other than their own incompetence and inadequate infrastructure. No doubt it's the evil Chinese, or those god-hating Russians again...
SANS is rapidly losing its relevance. At one point, they were a worthwhile place for training, but as their desire for more money grew, they apparently felt that FUD was more effective advertising than quality and word-of-mouth. Paller isn't the only one within the orgainzation that does this, just check out Ed Skoudis' presentations on ultra-mega-superworms.
Are there security problems out there? Sure. Should companies be thinking about them? No doubt. Are the bad guys routinely receiving multi-million dollar ransoms from power companies in ways that are untraceable and don't show up on the SEC filings? I strongly doubt it.
@airgap: "Air gap - The simplest security solution is usually the best."
Yes, but its not the most convenient way to run a plant.
When the senior management insist on having real-time plant efficiency reports available in their copy of Excel, the air gap gets replaced with a router/firewall. Not because the engineers think that's a good idea, but because the pointy-haired boss orders them to do it.
Maybe it was just the matter of how we interfaced, but every Fed I met while serving in the military (FBI,CIA, etc.) would likely lie more than tell the truth about almost anything. Even silly stuff, and too often we found out lying about big stuff. They turned out to be bigger threats than the folks on the other side of the border. It seemed it was more a joyride for them to see if they could cause us more trouble over improbable/impossible threat scenarios than anything resembling intelligence in any sense of the word.
I doubt Tom Donahue was lying. I think he has some information leading him to believe what he said. He obviously did not supply sufficient proof to satisfy some folks. But maybe that wasn't his intent.
He also apparently did not provide enough information for anyone to take action to defend themselves. He may not have had enough information to do that.
The intent behind the announcement seems to have been to motivate utility companies to assess the vulnerability of their systems to internet-based attacks. I don't have a problem with that.
A power outage in a major city? Don't forget many 3rd world countries have power outages daily. I frequently visit Cap Haitien, Haiti, which is a city with a population of half a million or more. They have 3 diesel generators supplying power to the entire city. I'm not sure, but what if these generators were being controlled by a Windows 98 computer? If someone hacked in and disrupted the power, would this be a story at all? Not to the residents of Cap Haitien who sadly have to live with unreliable government services. There are thousands of other similar cities to choose from. I'm sorry to say, but until more information comes out (hopefully FIRST to those protecting these systems) this is a non-event.
Power companies are so heavily biased toward operating cost reduction that they skimped on safety requirements in nuclear power plants and were thus the major contributors in setting back the nuclear power industry for thirty to forty years. Why wouldn't the same phenomenon apply to safety of their SCADA systems?
The problems with SCADA networks are the same problems with other types of "secure" networks. Unless the networks are segregated they are not really secure.
Even firewalls won't help if an internal host is hacked and that internal host has critical access rights to the SCADA network. The two networks should be completely isolated from each other. Period.
And, as one commenter pointed out, it's rather trivial to go to a substation, "hack" into the system (using the all-too-commonly used default passwords) and disrupt the network anyway. Connecting it to the internet merely compounds the problem.
Being in the audience when this was "disclosed", I have to say no one seemed to care. The CIA report was bland, lacked any new information and merely put me to sleep. How this become a blouted, our of whack, chicken little blurb is beyond me. Wish Alan Paller hadn't sensationalized the issue - takes away focus from the real problems facing Critical Infrastructure.
I agree with the FUD characterization of the CIA announcement. Thank you for that useless information. It was obviously designed to boost support for FERC's newly ratified Critical Infrastructure Protection cyber security standard.
It's important to note that this was from the CIA and referred to regions "outside the United States." Even DHS was caught off guard.
The squirrels are turning to blackmail? Terrible, just terrible.
I was at a KEMA conference a couple of years ago where a lady from Italy was explaining how they were trying to protect all their substations that they had hooked up to the internet from hacking attacks. The sites were protected only by user names and passwords.
Substations+Internet=Extortion ? Likely. I have heard rumors there are a few stations like this in the US, in fact I've heard rumors of industrial equipment shipping with Bluetooth. Im glad I have not been able to confirm this anywhere, hopefully the manufacturers have been clued in by now.
While I really hate vague security announcements it probably is enough to keep that next top manager somewhere from overriding security staff on the business reasons for enabling something like password enabled Internet or Bluetooth on an industrial device.
What Nick Weaver said. Tom Donahue and Ralph Logan are not apparachniks or bullshit artists.
maybe the cia are being somewhat vague because they trained up a few impoverished third-world dissidents, whom then stopped playing ball with their cia handlers - sorta like the afghan mudjahadeen - and it's an imminent threat.... help me out here, i'm trying to capitalise on the hollywood writer's strike, and i think i almost got the next mutha-frickin-snakes-on-a-plane/ghost in the shell/sleepless in seattle cross-over blockbuster. ;)
The extortions referenced were all overseas. First heard about it about last February. It has been attempted in U.S. unsuccessfully from what I understand. That person would not give me more details at the time.
Yes many SCADA systems in use and extremely vulnerable. I know more than a few that today run on NT4.0. I still see control systems projects that have expected lifetimes of 10+ years with minimal upgrades.
A full upgrade of a control system will typically require a 7 day minimum outage which will only happen every 4-6 years and only
- in the spring or the fall
- the weather is right (allowing loss of generation)
- not too many other power plants owned by the company are down
- needed highly specialized labor is available
Most companies do the simple basics right. The people in the field maintaining these systems are generally not even security amateurs though they do understand high availability very well. The engineering side is generally outsourced to engineering companies that could care less about IT security (at least until the recent CIP requirements).
We need professional control systems engineers with real solid computer knowledge to go with their PEs. The good companies bring in IT security experts in a support capability though no company I know of would ever let IT manage any part of a control system. I only know of a very few PEs in power that would be capable of changing careers and being a general IT security specialist.
Isn't this old news? The Register had something on it in 2005
And you were interviewing on the subject in Nov 2005
I went to the SANS/NISCC (now CPNI) Top 20 presentation in Nov 2006 where the same thing came up, but it seems as though there isn't any more publicly available evidence available than there was back in 2005/6.
I know a number of utility companies and their SCADA systems (unfortunately all outside of US). What is the nature of these networks? Out in the field Remote Terminal Units measure data and send it by hierarchically organized networks to a control center, which is sending commands back. Many SCADA networks ares till running proprietory protocols depending on the vendor, but more and more are changing to standardized IEC protocols.
All companies I ever visited are running at least one - geographically distributed - failover control center. The more important (backbone) parts of their SCADA network run on own cables and have fallback solutions like GPRS. The servers are redundant and have no direct internet connection. Attacking here wouldn't be so easy. But an attack out in the field wouldn't be difficult. And there internet connections are used quite often. Lost data and false values in these parts of the network are not very unlikely and the systems are able to handle this, so attacking is not fruitful and can only affect small parts of the network.
I do not believe in the danger of attacking SCADA systems by internet attacks. It is possible, but quite difficult compared to the result. It would be much easier and effective simply cutting some overland lines. You should be more afraid of asteroids falling on your head than power outage because of internet attacks.
there seems to be some talk in some circles of this happening in quite a high profile incident...
The IDF's C41 Corps ( http://dover.idf.il/IDF/English/units/forces/... ) are said to have shutdown both the Syrian electrical grid and the C&C systems of the Syrian air defense radars in support of last September's bombing of an undisclosed site of military significance within Syria's borders.
I'll see if I can't dig up something more solid than vague innuendo...
Mr W: Realtime efficiency/performance Excel tables can be retrieved even with keeping the air gap. A single-direction RS232 line from the secure zone out will do the job. Same can be achieved with an Ethernet cable with the receiving pair missing/disconnected/cut, and sending data out via UDP or raw frames. It can be done and it can be done on a shoestring budget.
Some of the comments out of SANS recently are so pathetic.
Terrorists blow stuff up, they don't email.
Many argue that ongoing electric sector automation developments have single-mindedly been focused on efficiency improvements to reduce headcount and get more out of aging systems. What often isn’t mentioned is how public safety and overall service reliability has also been dramatically bolstered over the years with automation- include distributed relay/plant protection systems with supporting operational network footprints that are also going more open systems with upgrades.
Electric utility industry senior management and boards are starting to get the message and get involved- still, there’s a need get more serious about making sure sufficient program support is implemented to effectively manage this risk domain and prioritize investments (people, process, technology) – including physical security. Top level support reviewing and helping improve cyber security will be a key part of meeting future opportunities with ever growing automation footprints and further internal business and customer integration developments (outage management, smart-grid, etc). Sites, such as http://www.digitalbond.com , provide a wealth of perspective on critical infrastructure protection and related industry developments. “Air gaps��? and “data diodes��? are great where they make sense, but there’s been plenty of compromised systems with problems introduced a lot closer to the action (w/laptops, USB sticks, human performance issues, etc). Putting the pressure on suppliers and vendors to improve solution security is also a key consideration, non-standard methods of changing passwords, maintaining software, etc make the job of just addressing basics even more difficult. Again, a more managed, programmatic approach to ensure a full range of risk informed appropriate controls are established, effective and continuously reviewed/improved is key- no easy, simple fix folks.
Commercial nuclear generation also has it’s cyber security challenges with the bar raised even higher; the big plus is having existing culture, plant programs and procedures that are much more amenable to cyber security program requirements being baked in using NRC endorsed NEI 04-04 guidance. The electric sector as a whole will continue to have gaps even with the recent FERC approval of NERC CIP to help move forward (good decision). Less anyone think that FERC/NERC has congressional oversight satisfied, check out what some have called a “train wreak��? of a congressional hearing from Oct 17th 2007 - “The Cyber Threat to Control Systems: Stronger Regulations are Necessary to Secure the Electric Grid.��? Subcommittee on Emerging Threats, Cybersecurity, and Science and Technology (Video/testimony Link: http://homeland.house.gov/Hearings/index.asp?... ) – with Mr. Joe Weiss providing his critical view of regulatory structure/CIP standards inadequacies. As for solutions, his stance that the CIP standards should just be scrapped didn’t makes sense to me; however, his points emphasizing NIST and other standards going forward does make sense and is reflected in FERC recent approval of CIP standards. Additionally, FERC directed the ERO (NERC) to make specific improvements– removing some compliance wiggle room, making the standards more effective. (Link: http://uaelp.pennnet.com/display_article/317382/... )
I was surprised to see that the Wikipedia List of Power Outages (wide scale) attached to the 15 Feb 08 Crypto-Gram did not include a whopper in New Mexico in 2000. This one was the unintended consequence of an act of arson. On 18 Mar 00 unidentified arsonists set a grass fire near Farmington, NM that generated enough smoke to cause two 345 kv transmission lines from the 1,780 MW San Juan and the 2,270 MW Four Corners power stations to go down. Power switched to a 230 kv line, which overloaded then went down. The resultant power outage affected virtually the entire state of New Mexico, and extended into the area of El Paso, Texas. An estimated 1.3 to 1.4 million people had their power interrupted a little before 1700 hours. Power was gradually restored from south to north between 1800 and 2000 hours. By 2000 hours about 50% of Sante Fe was still without power: Albuquerque had about 80% of its power restored by that time. It was the largest power outage in New Mexico history.
I work for an organization in the nuclear power sector.
I can't give too many details, as an investigation is ongoing.
But about a year ago, we detected significant attacks against our network.
We met several times with the FBI. As we culled through the information with them, I began to get the impression that:
1) this was not an isolated incident - they had a spreadsheet of several hundred IP addresses that had been generating traffic in "related" attacks
2) that many power/industrial facilities were being targeted
3) that there was substantial belief that the attacks were "state sponsored"
The attacks against us stopped pretty soon after they started - perhaps the perps looked at our website and realized that we don't have any real connection to any critical infrastructure, but I assume they were hoping we did.
Perhaps this is an isolated incident.
But facts are facts - SCADA systems have been repeatedly compromised. And industry seems reluctant to do anything about this.
If you can take a few power producing facilities off the grid - you can cause widespread, highly visible inconvenience. If SCADA systems are easy to compromise - then the likelihood of an attack is increased.
I can't imagine that these things can't be secured...
Then again, looking at the economics of it, clearly if a company thought that the potential lost revenue from having their name smeared across the front page of the Post outweighed the trivial cost of upgrading their SCADA infrastructure, they'd to it, right?
Perhaps, but not if they are focusing on more visible - but useless - 'security' improvements.
I used to work at the manufacturer of the world's most popular RTU in the electric industry. It has a massive security vulnerably, to wit: a backdoor password that is hard-coded into the firmware.
These units are routinely attached to POTS phone lines, or to TCP/IP networks.
The entire industry is stuck in the dark ages insofar as information security. Many electric utilities don't even have properly trained technical personnel to configure RTU's in the first place, nevermind actually oversee, with professionalism, their integration into an overall security framework.
It's an accident waiting to happen. The whole industry is, for the most part, extremely complacent.
I wish we had more individuals speaking from an industrial position here.
Security starts with your control strategy, and there should be two elements at a minimum, perhaps three if you need to share data externally.
Your process safety system (PSS) is distinct from your process control system (PCS). One protects the process from operating outside of safe parameters, while the other is used for process management and normal operations.
Your PSS is a hardwired system. It uses a deterministic communications channel - Not Ethernet - that has safety certifications, such as ControlNet or Profibus. It communicates with with no other network. When a process variable goes outside of safe limits, the process shuts down, contactors open, drives brake, pipes and vessels blowdown.
PCS is just for the money. PSS is to protect lives and the environment.
Process Control Systems (PCS) and Process Safety Systems (PSS) are tw
Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc.