Hacking Ukraine's Power Grid
This is an excellent article on the December hack of Ukraine’s power grid.
Comments
@K.S. It only seems stupid if you imagine being connected to the Internet means www – a webpage kids can beat on with passwords until they get in and start fiddling with controls to a nuclear power plant. That’s what they show on TV.
Matt • March 9, 2016 9:18 AM
Depends what you mean by “on the internet”, they didn’t have public IP addresses or interfaces that were publicly accessible.
According to that article they were on a dedicated SCADA network isolated from the corporate network with a firewall boundary. SCADA access was using a VPN client on the corporate desktop connecting to a VPN endpoint on the edge of the SCADA network. The attackers spent a lot of time infiltrating the corporate side to harvest those VPN credentials.
Anthony • March 9, 2016 9:43 AM
Personally I’d say that “on the internet” means not airgapped from the Internet.
JAH • March 9, 2016 9:47 AM
@Anthony +1
K.S. • March 9, 2016 9:49 AM
I intended to say that this isn’t in the same category as Stuxnet. The fact that control network wasn’t physically isolated makes this less impressive.
Clive Robinson • March 9, 2016 10:37 AM
Hmm, why am I not surprised…
I used to be involved with the petrochem, power and telco industries. Back last century I was warning people not to connect SCADA systems that had no security to communications networks without adequate security on the devices, the least of which was totaly air-gapping from public networks. Another was that the security had to be at the devices –link encryptors as well as end to end encryption– such that even if the air-gap was crossed an attacker was kept out of direct control of devices thus forcing their point of attack towards the center where better security supervision and control could be put in place.
One aspect of this is a “F##k Button”, you find in ordinary broadcast systems. In essence it puts a small time delay in between an incoming call and the transmitter. If the caller uses any “naughty words” the broadcasters staff hit the button and what was said does not make it to air. You can design nearly all industrial control systems such that on “loss of control” they “fail safe” or to predetermined conditions. Further you can in the process make them non time critical in human terms, which is about a ten second delay in most cases. Thus if a cut out button was pressed by an operator to disconnect comms from the actual supervisory stations the attackers would have found that there attack would have had minimal impact.
As for Firewalls and much other security technology, they are a thirty year old technological cludge that in reality has only changed in small “baby step” improvements. Thus classes of attack from shortly before 1990 still work…
To put it bluntly we needed to up our game back in the 90’s, but we did not in comparison with attackers. This failure can be attributed to various causes such ad Microsoft failing to secure their OS adequately and making it difficult for others to do. Software coders and system designers trying to go for the “new thing” not the “secure thing” (for amongst other things “C.V. polishing”). Oh and behind that managment making cut backs on staff etc etc etc for short term profit over longterm availability…
Thus all of this was almost entirely predictable and probably preventable.
As for who to blaim, it’s unlikely that there would be enough evidence for “on balance” let alone “beyond reasonable doubt”, and if there is it’s unlikely to see the light of day.
I would suggest that yes it does fit Russian Political MO, but that in no way indicates guilt. Further Russian Ownership interests would be served by “minimal damage” as would Russian Political interests.
But in the “smoke and mirrors” of “the great game” what appears obvious at first sight might not on a second look, but then might again on a third look etc, etc.
I can think of many ways of making it look like one thing whilst in actuality be an entirely different thing. And I’m sure there are other people around with a great deal more knowledge of how to do this than most of this blogs readers.
All you can do is look for who benifits most not at the end of the day but a few days weeks or months into the future.
For instance what would say Germany get out of this, or the UK or US or China or perhaps Israel or Iran. If you take the geo-political view, Israel for instance does not want Russia with easy access to the coast of Eastern Europe or the Middle East. If you look at geo-energy, China and Germany would gain certain advantages as would other Eastern European nations. As for the UK and US, stiring other peoples pots is what their IC and MIC live for, give Russia a cold in Europe, and it might sneeze in Syria etc. Then there is Iran, they’ve shown in the past they have the skills, what would be the upside of keeping the US, UK, Israel and several others busy.
If we are talking of “sending messages” Putin could be doing it to Russian business men who are perhaps seen to be straying from the fold. It might even be rival Russian businessmen. It might even be the Ukrainians doing the same thing. It certainly has the signs of being a message but to whom and from who.
As an attack it was actually a failure in that it did little or nothing but be an anoyance, and wasted months if not years of work. Further it’s also ensured that a lot of open doors are going to get found and closed.
One of the interesting tells is the “0ld Skool” attack vectors. If people have a hunt on this blog you will find one or two people have warned that todays “youngsters” are not learning from what you might call Cyber-History. This suggests that some of those around this attack have been around for quite a while. Thus they are not as likely to be cyber-criminals etc and almost certainly in their fifties. This, suggests that they have “protected status” of some form, and could actually be from what most would assume is the “White Hat” or Security Industry…
But remember it’s all as much of a guess as the next persons.
I’d hesitate to over-blow the merits of airgapping as an isolation mechanism – although I think it should be mandated unless the organisation really does have its act together.
For one thing, delay in accessing the systems, or failing to being alerted to problems, could act as a service disruption in its own right, regardless of hackers. Three Mile Island scenarios come to mind.
For another, airgapping isn’t foolproof, and might lead to complacency – after all, people are still operating the things, and they might not all be trustworthy.
The thing that’s struck me about this is the necessity for two-factor authentication, and better operational security. After Belgacom, it certainly makes sense for operator control systems to be rigorously banned from doing “normal” stuff, including anything that’s vulnerable to spearphishing – which was apparently used in this attack. Often, it’s a combination of penny-pinching (not providing distinct clients and segregation, not running two-factor), and operator carelessness in combination that opens the holes.
Terrence • March 9, 2016 11:15 AM
Many years ago the Navy came to me and asked what could be done to render enemy weapons useless so I devised a system to do that and provided them with a button they could push to immediately deactivate all enemy weapons everywhere. They said it was so ingenious, what followed were numerous calls from the banking industry. I told them how stupid they all were so they hired me to fix all their problems. Buy my book.
Scott • March 9, 2016 11:22 AM
@Terrence
That’s nothing. I started a company that catalogs anything that goes wrong. When people come to us with a problem, we just record it and then tell them “you shouldn’t have done that.” Now, the company is worth millions.
Mr Peabody • March 9, 2016 11:44 AM
“For another, airgapping isn’t foolproof, and might lead to complacency – after all, people are still operating the things, and they might not all be trustworthy.”
I think this is a reasoning fallacy; it’s sacrificing the good in search of the perfect. Failure via complacency in this context would mean that a single attack at a single location gets through. Failure with sites which are networked together means a single attack takes down an unknowable number of locations. There’s a big difference between those two scenarios.
The facts are in WRT to network security- it’s imperfect because the defense’s technology is not qualitatively different from the offensive technology used to break it. It will always be a game of cat and mouse. Network computers are subject to all attacks air gapped computers are subject to PLUS network attacks PLUS zero days inside of the 3rd party software they run on.
It’s not just that air-gapping computers works to thwart nearly everyone outside of state-level actors (well, so far) it’s that air gapping computers importantly and dramatically narrows the vector of attack. What’s left to the attackers looks like sophisticated electromagnetic radiation attacks (let’s include sound-wave based attacks here) and physically connecting a device the attacker has controlled at time point one to the machine to be attacked at time point two.
Between a sprawling confederation of networked computers and a series of individually air gapped computers which set of castles would you prefer to defend?
It obvious.
p. puk • March 9, 2016 12:26 PM
Can’t read the article without whitelisting addblock… Not something I’m willing to do.
wired adblock • March 9, 2016 1:02 PM
@p.puk
this rule worked for me:
|http://www.wired.com/assets/load?scripts=true&c=1&load%5B%5D=jquery-sonar,wpcom-lazy-load-images,outbrain,blockadblock*
Steve D. • March 9, 2016 1:13 PM
@p. puk
Just disable scripting on the page. I use Firefox Accessibility Extension v1.5.61 and just Disable Scripting and reload the page when I run into something like that.
Tatütata • March 9, 2016 2:24 PM
Clive,
You can design nearly all industrial control systems such that on “loss of control” they “fail safe” or to predetermined conditions
How do you add a 10s delay on a reactor SCRAM control?
Or immediately shed load after loss of lines or generation capacity?
Or rebuild your grid after a collapse?
MIA Paper Planes • March 9, 2016 2:27 PM
@Clive Robinson
If people have a hunt on this blog you will find one or two people have warned that todays “youngsters” are not learning from what you might call Cyber-History. This suggests that some of those around this attack have been around for quite a while. Thus they are not as likely to be cyber-criminals etc and almost certainly in their fifties. This, suggests that they have “protected status” of some form, and could actually be from what most would assume is the “White Hat” or Security Industry…
Being an “old school” hacker, who posts here fairly regularly, I am not sure what that has to do with the Ukrainian plant hack? Reality is most of the good hackers were security researchers. They were making money then, and make money now. There is simply no motive to strike out at an Ukrainian power plant.
Also, a professional hack is a team job. And this job, I completely agree with the assessment made, it was military in planning and execution.
Key terms there: military in planning and execution.
It is one thing to know how to find vulnerabilities of substance. Something else to be able to write custom, high grade malware. Something else to weed effectively and efficiently through a power plant.
And something entirely else to first create a team, manage a team, gather intelligence on the facility operations infrastructure, and create plans which are realistic and workable to execute action.
The later you see only in military and high level intelligence operations. There, it is standard, and an art form.
Which is exactly what this hack has hallmarks of.
I can further state that Russia has long had a penchant and expertise in infiltrating power plants and really, anywhere and anyone, in the fields of energy.
As you well know, Russia also has a very long history of respect for the strategy of quiet infiltration and silent waiting.
I disagree with the author’s assessment that this is unlikely Russia. It is highly likely to be Russia. Not just because of “political MO”, but because of technical MO.
On a basis of good, even ‘best practice’ investigative principles, it is always best to keep everything open. Settle on nothing. Gather evidence, stick to evidence. The evidence I list is – hat tip – circumstantial. So it could be anyone else. It is very true one nation state can easily “fire” on another nation state, as attribution so often relies on circumstantial evidence. And the technical details are hard to fathom.
(For anyone outside of the field, who do tend to be the consumers of intelligence.)
My read on the article was: what was the real intention here? Is there a target they have in mind, who is supplied by that power station.
It should not need to be noted that Russia is very advanced in electronic warfare, which they have been using to scary efficiency in the Ukraine.
They are highly efficient in jamming technology, and very aware of the advantages control of power in regions they may wish to run operations in could be highly advantageous.
The ramifications of manual circuit breaking, to that end, I am not sure.
It might be noted if a target installation even does have backup power generators, forcing it to use those removes one big potential problem from the equation for infiltration.
(Even though this can often be done by more simple means then disabling a region’s power plant. But, the resulting chaos from such a scenario could be highly conducive to pulling off all sorts of operations. Shutting down police and ems. Keeping a strong diversion in the area.)
Strategically, they blew their hand. However, this very well could have been more valuable as a test shot, if they have other means of getting inside.
As for American “protected hackers”, I think that is not how things play.
There may be covert operatives playing as hackers. But, the US has plenty of well trained teams ready to run operations at anytime, anywhere. Having uncontrolled rogues running off unplanned operations stateside is dangerous business.
Only reason someone like “Jester” has not been pulled in or stopped, is because it is not worth anyone’s trouble.
The legal infrastructure even for prosecuting domestic hackers remains poor.
I will suggest, technically, they did a very good job wiping the systems. That is typically performed not for the function of disabling the systems (as the firmware attacks appear to have been used for), but to ensure there is minimal forensic data left.
The cyber equivalent of making absolutely sure there is no blood splatter or bullet casings.
But, again, that is military or intelligence MO.
Clearly.
@Mr Peabody – re. airgapping,
No it isn’t obvious for your scenario, because you’re ignoring other downsides including accessibility and response time. The scenario you’re envisaging seems to be individual control terminals co-located or somehow directly attached to the controllers. This is not a maintainable scene for a large-scale utility.
Alternatively, the control terminals, control network, and controllers could be airgapped (maybe!) to allow a central control centre. And that is a more defensible scheme, but loses you the ability to be notified in case something goes wrong (unless you’re assuming 24 hour coverage at the central control center). And, realistically, you have to get updates in and data out, so unless that’s all going to be done on usb sticks or something, there’s still a substantial deal of awkwardness in all this.
Seems to me that the cardinal sin is having the control terminals situated on the bad side of firewalls and maybe fulfilling multiple functions. And no 2FA.
jim • March 9, 2016 3:37 PM
I’d like you to tell Wired that I get it, they’re in business to serve ads (and scripts from places I don’t even recognize – yes, I peeked). I’d subscribe to an adless and scriptless version, but I’m not going to give my credit card number for an automatic monthly charge to anyone unless I know what those scripts are.
Bob Wire • March 9, 2016 3:43 PM
Unfortunately, there are very few with in-depth ICS and information security knowledge/experience and most of them are squirrelled away in obscure cubicles of energy companies. Many would benefit if their skills could be tapped in such investigations.
It is a well written article, but one clarification: The term “grid” in power systems typically refers to interconnected transmission facilities. This event happened on the distribution level.
Buck • March 9, 2016 6:51 PM
@de
The scenario you’re envisaging seems to be individual control terminals co-located or somehow directly attached to the controllers. This is not a maintainable scene for a large-scale utility.[citation needed]
Surely you could provide us with a recent cost-benefit analysis of this!
Dirk Praet • March 9, 2016 8:54 PM
@ de , @ Mr. Peabody
Seems to me that the cardinal sin is having the control terminals situated on the bad side of firewalls and maybe fulfilling multiple functions.
Although completely airgapping the business/IT and process control network (PCN) is the more secure (yet not bulletproof) solution, it’s also cumbersome in terms of management, monitoring and remote control. Hence most facilities, especially larger ones spread over multiple locations, eventually adopt some kind of (routing) firewall implementation between them.
Whereas the lack of 2FA in this particular case is an obvious fail, the cardinal design sin in many such SCADA networks is that once passed that firewall the entire PCN is treated as one security zone. This is a typical late nineties, early 2000’s border defence strategy. Over time, all sorts of firewall rules get added to facilate questionable stuff and unless you have a truely fascist firewall administrator who keeps a tight shop (and documents everything he does) it eventually turns into a Swiss cheese.
A more sensible approach consists in further network/access segregation of end devices (PLC’s, RTU’s and IED’s), SCADA servers & operator consoles, and PCN access by means of distributed firewall bridges that also offer DoS protection, IDS and SCADA protocol filtering.
That said, it was an excellent hack, well prepared and professionally executed. Especially the re-writing of the firmware on the serial-to-ethernet converters is a master piece. Props to whatever team pulled this off.
fajensen • March 10, 2016 4:08 AM
Were I work, we have a discussion between “Bright Young Things” who want to store PLC code in a repository and upload it remotely to the PLC’s and “Boring Old Farts” like me, who don’t mind the repository at all because version controls are good – but – all believe is it much simpler and safer to send a techie out to the plant with an EPROM module containing the new code if anything needs upgrading. PLC’s are rarely updated, so there is, IMO, no good reason to be clever with it.
The “Bright Young Things” will probably win the first round – because management does not really get that when they pay good money for expert advice, then, they should listen to that advice even when it is not what they like to hear.
The B.O.F.s will, however, win the one round that matters – the one held at the inquest.
Clive Robinson • March 10, 2016 5:15 AM
@ Tatütata,
How do you add a 10s delay on a reactor SCRAM control?
Or immediately shed load after loss of lines or generation capacity?
Or rebuild your grid after a collapse?
Hmm three questions 🙂
There is a big difference between control types. “RED” or “Emmergancy” control circuits by definition should never be part of a “programable” path or one that can be easily disrupted adversely by definition.
In the past this involved building fail safe state machines in old fashioned “ladder logic” as directly adjacent to what is being controled as possible. This gives a minimum loop time in whatever technology is in use. Sometimes you do require external input from other places, this should be by old standards “hardwired” on individual circuits. Thus a full shutdown has one circuit, a pull to fail safe another circuit and so on. The fail safe and shutdown circuits should never have the ability to be over-ridden remotely. The down side of such systems is “capacity” setting, in the case of a generator, there is a lot of difference between short term peak output and longterm safe mean output. Setting ladder logic to deal with operating above safe mean has significant reliability inplications, however not doing so has significant plant cost implications. The former is a longterm implication the latter an upfront or short term implication. It’s not difficult to work out where stock holders thus C level managment want things, and likewise cautious engineers. Thus the trade off via programable systems, which in turn leads to a cascade of choices which have put us where we are.
Have a look at the report into the NASA STS booster failure of an expanding “O-Ring” that caused burn through and shortly there after the loss of vehicle and crew one of whom was a school teacher within seconds of take off. NASA is still reputed to be one of the most safety conscious organisations in the world, so it’s not diffocult to work out if “cascade risk degrading” effected them what is happening in other organisationd with different primary motivations.
As for “re-building” post a colapse, there are a number of considerations. Even if no equipment harm happened during the event you can not simply throw the switch to bring everything back up due to the likes of “in rush load”. A simple example is the old hot filiment light bulb. When the power goes off it’s switch stays on the filiment cools and it’s impedence drops to a fraction of it’s hot impedence. Thus the cold power load is many times that of the hot power. Under normal conditions this is a quasi-random event thus the transient cold load of an individual bulb is lost in the noise. Not so when the generator set has to come up because those cold loas transient loads are in synchronisation thus trying to bring the generator up woth ten times it’s working mean load is a non starter. Which means “phased” bringing up of the network. If you want to know more the recent hurricane that hit NYC that took out rather more than the power grid has reports on what they did. One thing that comes through was that it was “A Grace of God” incident, because they had had an incident shortly before that which had caused them to change their mind about maintanence and capacity, therefore the network was not just more robust, they also had parts available without leed time or manufacturing setup time.
If you want to go into more specifics which are a little off topic, can you wait a few days for the thread to quieten down? Then I’d be more than happy to go into more specifics.
Clive Robinson • March 10, 2016 5:49 AM
@ MIA Paper Planes,
As for American “protected hackers”, I think that is not how things play.
Err I was not talking the specific of “American” but the more general of Any State who might have reason to take advantage of an oportunity.
My view is that a “Grey Hair” was involved either directly or as an advisor / contractor. There are three sources for such people,
1, Senior ranks of MIC.
2, Prison / past offenders.
3, Senior Researchers / leads in legitimate security organisations.
Because that is where the knowledge and expertise for the older style hacks reside. The first group like LEO’s have “protected status” already, thus it’s a non-issue for them. The second group are going to want safeguards as well as something in return, the least of which is freedom, status and comensurate employment. The third group are getting very touchy of late, as “bad publicity” has been a bit of a death sentance for their parent organisations to sign. This kind of started out with the outing of HB Gary, various other organisations that sold HackTec to represive regimes and more recently various AV vendors and even those regarded as “good guys” like RSA. But quite recently “taking grant money from the FedMan” to research how to break ToR and other anonymity systems has caused even academics to tread wearily.
Non of the above is an issue in Russia that has “protected status” laws even for murders (see Litvenenco) working on the whim of the state. Unless of course they want to travel abroad. Similar applies to Israel, and it’s reasonable to expect of China, Iran etc.
It’s the WASP nations where the problems bite the hardest morals and ethics and a need to appear “The Good Guys” even when they have blood to the elbows and beyond, has a chilling effect, especialy as trust in such governments wains and the citizen’s start “looking behind the curtain”, to find those to blaim, even old style IC Contractors are hearing “the wheels of the bus” heading in their direction, and are no doubt taking precautions they are not a JFP under them.
Dirk Praet • March 10, 2016 7:11 AM
@ fajensen
Were I work, we have a discussion between “Bright Young Things” who want to store PLC code in a repository and upload it remotely to the PLC’s and “Boring Old Farts” like me
I prefer the term “Grumpy Old Men” (GOM). The “Bright Young Things”, as you call them, in general haven’t yet experienced the infinite number of ways in which blind adoption of shiny new technologies can go horribly wrong. Which unfortunately tends to cloud the vision of both groups. And that’s where meticulous risk management steps in again.
albert • March 10, 2016 10:08 AM
On a scale of 1 to 10 (with 1 being FOX!News, and 10 being a peer-reviewed academic study), Wired always seems to rate around 5 for the amount of factual content. So there is a lack of data for this incident. Windows seem to be involved, no surprise there. A guy opens a Word file in an email attachment. HTF does that happen on an HMI in a critical control system?
Unlike nuclear power plants, which can contain all control system communications in one physical area, electrical distribution systems cannot do so. They often cover great distances. In the good ole days, -control- systems used dedicated current loops, RS422 serial lines, or leased telephone lines for station-to-station communication. Today, it’s the Internet.
At last computer job I had, we used PC Anywhere to remotely control the PC HMI in our machine at the customers site, 1200 miles away. We could also remotely program the PLC. Pretty cool! Today, it’s built in, and more dangerous. You can set up distributed control systems with PLCs and do all your Input/Output through Ethernet.
Stuxnet taught us that air-gapping isn’t a magic bullet, but that doesn’t mean it shouldn’t be used. In nuclear power plants, reactor controls need to be air-gapped from the world, including the power distribution center within the plant itself. Control systems in all power plants need to be air-gapped from the world as well, but this is a problem; there’s too much integration. The only external control a power plant control system needs is how much and how soon. These instructions come from the various ‘control centers’ around the country. These are located in anonymous buildings and are fairly secure physically. The engineers are in constant communication (by phone for critical cases) with all the centers within their service area, and often other sectors; basically any region that can provide or consume power to/from your region. This system needs to provide fast and accurate information, and being distributed on the Internet, it’s a major security problem.
That said, there are simple things that can be done to increase the security of control systems. For control system computers(CSCs):
- Air-gapped from the world. Physically secure internal LAN only, no routers, no wireless.
- Get rid of Windows. Bare minimum OS to run HMI software only.
- USB/CD/wireless/etc. capabilities permanently disabled.
- PLCs can be programmed from the secure LAN only. No local access (at the PLC) allowed.
- No portable computers/phones of any kind allowed in the secure areas.
The problem of verifying PLC/PC software is another nightmare issue, especially when the current programming paradigm is: “we’ll fix it in the next update”.
Distributed systems need to dis-integrate the business side(BS) from the control side(CS). I don’t think VPNs are the answer, when you have a system where -every- PC is a potential virus incubator, totally dependent on human operators for security, and totally exposed to the bad guys.
I didn’t address data acquisition/monitoring issues…
. .. . .. — ….
MIA Paper Planes • March 10, 2016 3:50 PM
@Clive Robinson
Very engaging and intellectually challenging response.
https://www.f-secure.com/weblog/archives/00002747.html
https://blogs.mcafee.com/mcafee-labs/updated-blackenergy-trojan-grows-more-powerful/
I had to look up the blackenergy malware, and right away discovered additional information about this attack.
That it was used made me feel uncomfortable with suggesting it could possibly be the work of a single malcontent security researcher.
The terminology everyone is using is “intelligence working with organized crime”, or similar.
What that really simply means is Russian intelligence working with Russian organized crime.
(China is known to work with ‘cybercriminals’, but it has never been suggested they do this with their organized crime, the triads. Russia, quite the reverse.)
My hunch is it is simply the GRU.
Maybe russian intelligence works with russian organized crimes in some attacks. But on something of this sensitive of a mission, I do not see how they could do such a thing. Especially as this attack reeks of being strategic – entangled in future ‘battle plans’ – and in the middle of such an extremely delicate and covert military operation.
I believe you are positing it is possible for an American or European rogue security researcher running such a false flag operation?
I do believe it is possible for any mid level consultant who has performed security assessments on power plants to take one down.
It is also possible for that action to be misunderstood as being more ordered then it was. So, you have the consultant, an ex-AF officer engaged in cyber military planning, who juxtaposes his own perspective onto the forensic data of the attack, and sees order and pattern… where there is none.
But, how did the attacker get the latest, ‘never before seen in the wild’ blackenergy malware? Never seen by major AV firms. They have pretty good coverage.
Very unlikely the attacker both performed this operation and updated the blackenergy attack malware.
Maybe a consultant who had previously been engaged in forensic work specifically at power plants, and so has malware analysis work could get an earlier version of blackenergy. But, highly unlikely they could update it and perform the attack.
Some corporate group like Kaspersky, who performed review of the trojan, could have been involved. But, then, that would mean Kaspersky has a group which is really very, very deep cover, covert GRU.
So, it would be GRU still.
The ‘experienced IC contractor’ sort going rogue because ‘being thrown under the bus’ or something, is certainly a very curious and interesting theory, however.
People with top secret clearance, pretty much have guaranteed jobs however. People with significant security assessment do too. Security sector is not one of the sectors where people are complaining.
Even if they were let go or something, their prospects would be incredible.
Definitely interesting theory though!
Maybe you have more thoughts on that and can flesh it out.
But, anyway, otherwise, just my two cents.
Interesting hack, but Russia has been hacking power plants all across the world now, and that for a very long time.
However, even in that context, first time they ever “let one loose”.
But, that is why they hack them. They are like their old stored weapons cache. “To use in case of war” ‘battle plans’. In this case, there is war, and they are using it.
I hope nations targeted by Russian GRU take good notice of this.
DoE, hope you are really paying attention.
Sure they are.
Key terms, flag. Quote.
😉
WhiskersInMenlo • March 10, 2016 4:49 PM
The obvious blunders in management make me want to dismiss this as
less than interesting breach. However the need for and impact of IPv6 will reopen
many risks some think solved. NAT and VPN are used to solve a lack of IPv4
numbers but shortly IPv6 will become necessary and NAT mapping and filters
will vanish and VPN routing less obvious.
Time for a lot of us to go back to school and think hard about this.
Good article … has me thinking.
Some guy with experience • March 10, 2016 6:54 PM
No 2FA is a significant blunder, though there are ways to defeat that if people aren’t trained or just a mistake.
I keep hearing air gaps. Air gapping is not a real option as instant changes across long distances are necessary. Prices would also at least double to go old school. Imagine the cost of putting a person at the 16000 US/Canadian bulk power substations (not including your local distribution stations) 24 hours a day. That alone is at least 70,000 people or at least $4-5 billion per year payroll not considering all of the other employee costs like equipment, training, benefits… Now add probably another 80-120,000??? distribution substations. Now try to find that many qualified people. Though a half million new jobs would make a dent in unemployment.
The 4 North America power grids (eastern, western, Texas, and Quebec) are themselves an interconnected network of power grid networks. And these are dependent on 9000+ generation units, 12 regional control centers and dozens of control centers in each region. If you try to isolate it down further, like Texas does with its in-state nearly isolated power grid to avoid pricing regulation, you create supply and demand issues on a system that must be balanced. Power cannot be stored and must be consumed and produced in balance.
This bulk power system does not use the Internet to run, only dedicated communication networks. Local distribution is a different story though everybody of any size uses a separate network here. Distribution grid impact is isolated unless a large number of distribution grids gets hit simultaneously. Even 3 companies in a country the size of Ukraine didn’t take down the power grid, which is actually an interconnected part of the Russian power grid. Smart grid, however, is a whole different Internet of Things story.
Isolating from corporate networks is also not reasonable. With the sheer quantity of information needed to integrate all of the separate pieces, you will massively increase the amount of time a person’s power is out in a year. Maintenance systems, repair systems, regulatory compliance systems, billing systems (customer and between power company billing), power purchase and sale systems in deregulated markets, and more need to talk to each other.
A true air gap is not affordable, not efficient, and bad for reliability. Proper security must be used. Adequate bulk power system capacity must be added to prevent cascading failures.
MIA Paper Planes • March 10, 2016 9:43 PM
WhiskersInMenlo wrote:
The obvious blunders in management make me want to dismiss this as less than interesting breach.
I would suggest the breach is interesting because Russia has been hacking energy companies, energy vips, and power plants for years with sleeper cell attacks. This is the first time they “activated” one of these attacks. Doing this in a very covert war zone also helps prove the case that this is the work of their military intelligence, not SVR.
If there are moles in GRU already providing this information, there is now multiple layers of cover so these subjects can be brought up more loudly.
Not an intelligence analyst. [Comp sec researcher, as noted above, with several decades experience.] So, maybe I am off here somewhere. The information about the attacks on energy companies and power plants largely have gotten over the years from socializing with peers in that industry and hearing their horror stories.
It does sound like the attack on the plant in Crimea incited this effort. Which may mean there are not strategic ramifications that may play out later. Which means it was the military intelligence equivalent of a premature ejaculation. They fucked themselves. They played their hand too soon.
Comp sec angles/Dirk Praet/Someone with experience:
Also not a network security guy. I have, however, observed missing in countless places. I am also aware, as the previous poster from the energy sector pointed out @”someone with experience”, that there are still ways around that. As a sec researcher, there will always be a way in.
But, 2fa is clearly the first major step they need to take.
While not a net sec guy, I do believe that @”Dirk Praet” is right, that you must have ‘behind the DMZ’, lateral movement detection. Strong visibility. While I agree that is 1990s security to rely on perimeter security, I disagree that there is ever a way to win by only perimeter security regardless of how fascist your network security architects and engineers are. I also disagree that good behind the perimeter security is routinely and roundly adopted. Not what I have been hearing from associates in that area.
I do absolutely agree that is the best approach to take, next to 2FA.
Would figure, but not sure, microsegmentation, strong visibility tools, strong heuristic systems designed for detecting the traffic patterns of malicious inside dmz movement, low false positive heuristic systems designed for lateral movement attacks behind the dmz, and strong blocking capacity are all types of technology which would be along the ways of the right and best defense.
Surely attack movement would be very, very different from normal usage movement especially in scada networks?
They are surely even in very busy and diverse corporate enterprise networks…
Messages:
It sounds like the US intelligence and protective services (DHS in this case) were all over that. I am sure the DoE will be as well. Hopefully, the US and other nations who have been experiencing these attacks from Russia really take notice and force change at critical infrastructure systems.
But, IDK, “not my area”. But, definitely an interesting attack.
IMO it is looking like while I do believe Russia’s SVR – and maybe even GRU – has been working with organized crime in intelligence gathering espionage, I think this is very likely to have been only GRU.
Why and how could they possibly pull off something like that without revealing who they are to their criminal partners?
Seems way too risky for them.
Any organized crime working with russian intelligence want to make more money then they will ever pay you? LOL.
Not a hard message to send out.
And sure they would get exactly that.
GRU is surely not that stupid.
So maybe the fingerprint of “unprofessionalism” in reconnaisance just sucks, or “just looks like that”. Reconnaissance is hard.
Definitely seen professional intelligence services use ‘off the streets’ hacker tools before. Gives obvious plausible deniability without having to engage in relationships with untrustworthy people.
(Also a lot of reconnaisance tools that are actually very good just happen to also be popular with script kiddies. shrug.)
(Havij, good example, or Burp.)
MIA Paper Planes • March 10, 2016 9:45 PM
^^^^^
“I have, however, observed missing in countless places. ”
read: I have, however, observed 2fa missing in countless places.
Buck • March 10, 2016 9:50 PM
@Some guy with experience
Imagine the cost of putting a person at the 16000 US/Canadian bulk power substations (not including your local distribution stations) 24 hours a day. That alone is at least 70,000 people or at least $4-5 billion per year payroll not considering all of the other employee costs like equipment, training, benefits… Now add probably another 80-120,000??? distribution substations. Now try to find that many qualified people. Though a half million new jobs would make a dent in unemployment.
$5 billion per year payroll is a good number for me to make some back-of-the-envelope calculations with. Thanks! The extra costs, (equipment, training, benefits) I’m not too familiar with… I can do more research to come up with a better estimate, but for now, could I just guess about 3x? So, $5 billion for payroll and $15 billion for new equipment and support/training would then bring us to $20 billion. Assuming that distribution substations will follow the same price curve (and rounding up to 160k of them for ease of calculations), multiply by 10, giving us a grand total of $200 billion per year. Wow! I’ll admit, that’s a lot more than I initially suspected. Yet, isn’t a stable power grid considered to be a critical infrastructure necessary for the common defense of the people and generally good for our global society?
If the following reporting from Reuters is accurate, we could’ve already had this implemented and paid for, for 42.5 years, (ignoring inflation) with barely anybody even noticing…
Special Report: The Pentagon’s doctored ledgers conceal epic waste (November 18, 2013
Because of its persistent inability to tally its accounts, the Pentagon is the only federal agency that has not complied with a law that requires annual audits of all government departments. That means that the $8.5 trillion in taxpayer money doled out by Congress to the Pentagon since 1996, the first year it was supposed to be audited, has never been accounted for. That sum exceeds the value of China’s economic output last year.
http://www.reuters.com/article/2013/11/18/us-usa-pentagon-waste-specialreport-idUSBRE9AH0LQ20131118
MIA Paper Planes • March 10, 2016 11:00 PM
@Clive Robinson, @subject of politically motivated hackers
I also feel I should add that while I certainly believe your assessment is chock full of interesting stuff, I have to also disagree about the viability of political motivated hackers.
While this attack could have been performed by a mid level security consultant with experience in performing security assessments on power plants, not even a ‘grey haired’ security researcher… that reason, from the sparse information we have, is simply the usage of blackenergy.
What I did not mention is I am not aware of any political motivated hacker who has ever been very technically sophisticated.
You mentioned Sabu. Sabu was a low level web penetration consultant at an average security consultancy. As far as we know, he was not under FBI control during the HBGary Federal hack, which was performed by social engineering. He certainly was during the Stratfor hack.
None of those details change the fact that neither he, nor any of the arrested party had any capabilities beyond “low level” consultant or script kiddy.
I mentioned Jester. Likely, a report about him is true, that he was some manner of computer security contractor in Iraq. While he is very smart and careful, none of his work indicates he has anything above low level computer security hacking skills.
China has supposedly had some sophisticated, political activist hackers. Supposedly. Reality is they do have a number of brilliant security researchers, but I have never met any nor heard of any who did not work either directly or indirectly for their government. Where indirectly is not under any pretext. Corporate security firms owned by the government, as all the employees well know.
Some of that story got started by the fiction of “Code Red”. They simply recycled well written exploit code into a trivial virus.
This fiction has also continued because China’s methods often appear amateurish. And they certainly have employed domestic hackers. But, they employed those hackers. And their amateurish methods have not only given them decent cover for very persistent, widespread attacks over many years, but have enabled them to actually do thos attacks.
A lot of recon is certainly stealth. But, much can not be. Ultimately, you are going to want to scan and test the perimeter and everything you can. Relying on precious, hard to find zero day and difficult to code rootkits is an obvious very poor strategy. Loss of such attacks is very painful.
Otherwise, what have you? Poorly performed distributed denial of service attacks. Clever, bold, social engineering attacks which succeed because only crazy teenagers would even try them. Sometimes reputation attacks, such as the attacks against the report Krebs, by organized crime — not political based.
Website defacing. Which has largely been out of style for a few years now.
@some guy with experience
“Adequate bulk power system capacity must be added to prevent cascading failures.”
Indeed – attempting to restrict damage via control systems highlights the situation that Nicholas Taleb warns against in Anti-Fragile. All our key systems seem to be built on reduction of contingency, common sense etc – because it maximises profit.
With the shutdown of old coal powered generation capacity, there’s wafer-thin margins on blackouts in any case, regardless of any attack – just need a windless cold day, and you’re over the edge.
Some guy with experience • March 11, 2016 6:42 AM
@Buck
Based on your numbers of $200B/year, that is $625/ per citizen or $52/month/citizen. My gut says it isn’t quite that high, but let’s use it for estimates. Hard to do quick estimates without more research.
Guessing 70% residential usage and 2.3 people per residence, the average home power bill would go up $85/month. That’s before businesses pass their cost indirectly,especially for large manufacturing power users (steel, oil…). There would be outrage if the industry adds a $1000 annually or 60% to your average bill.
In practice, simplifying the environment with less interdependencies across systems, better security practices, and better data hygiene would accomplish this a whole lot less expensively. Adding resiliency with more power lines so the grid is not so stressed in peak times would accomplish more, even if it’s in somebody’s back yard. And less Smart Grid that introduces massive security risks just to decrease outage response time and eliminate meter readers (a huge cost and one of the more dangerous jobs in the industry) would add a lot of security. Eliminating the grid efficiencies by eliminating coordinated grid management would have a much higher cost.
MarkH • March 11, 2016 10:51 AM
Clive’s notion of an automatic delay may be very useful, though obviously requiring careful judgment as to where it may be applied.
A dramatic example is the US land-based nuclear missile force. It is famously kept at “hair trigger” readiness, posing an obvious risk of [extremely] catastrophic accident.
One safeguard, is that if a Minuteman missile installation receives a valid but unconfirmed launch order, the missile will remain in its silo until either (a) 90 minutes have elapsed, or (b) it receives confirmation, at which time it will launch.
Throughout this wait, it will broadcast messages via multiple channels saying “I have an unconfirmed launch order” … giving time to countermand the launch order.
Automatic delay can’t be applied to actions which must take place as soon as possible.
However, it should be possible to identify a set of actions for which a delay will be predictably safe.
Also, and perhaps most usefully, hackers often make configuration changes to target systems as steps in their attack. These configuration changes are not operational actions, and in general are never time-critical. Why not have them follow a ‘Minuteman’ protocol?
MarkH • March 11, 2016 11:07 AM
.
ON THE QUESTION OF ATTRIBUTION
I cringed when one commenter (nothing personal) asked, who benefits from such an attack?
The “Cui bono?” (Who benefits?) heuristic is useful in identifying parties who may be motivated to take some action.
However, it has a F*CKING ASTRONOMICAL rate of false positives. Every day, millions of human actions and decisions lead to a variety of benefits to some persons or groups, and detriments to others. The great majority of these are unintended, and indeed unanticipated.
Cui bono is almost as scientific as saying, “there was a terrorist attack on November 19th, everyone who sent an email or make a phone call in the preceding two weeks must be considered a suspect!”
Cui bono is a favorite tool of wingnut conspiracy theorists (sadly, I know enough of these personally to speak from direct knowledge).
IT IS NOT A TOOL FOR THE FORMATION OF VALID ARGUMENTS. M’kay?
Let me suggest a different heuristic, which also has a lots of false positives, but which (I suggest) is hundreds of times more reliable:
“Who has a history of assault against the victim?”
In police investigations, this usually leads straight to the perpetrator (remember O.J.?)
Who has made an armed invasion of independent Ukraine? (Hint: there is only one answer)
Who has occupied independent Ukraine’s sovereign territory, and claimed it as their own? (Hint: there is only one answer)
In recent times, who has killed thousands of Ukrainian soldiers with their regular military forces? (Hint: there is only one answer)
MIA Paper Planes • March 11, 2016 5:17 PM
@MarkH
I do not believe that was me, as I cited a number of reasons of ‘why I believed it was most likely Russia’. I did not cite the many reasons I believe it is most likely GRU, as these are also obvious.
But, it is not just Russia’s history with Ukraine. Russia has been performing these sorts of attacks for a long time now against power plants. I can pull up old news stories, some has been reported. And the general trend has been admitted by US officials, anyway. But, most of my sourcing there is just from talking to peers who work in that sector. That typically at in person meetings, conferences, and the such.
Motive, means, opportunity.
Past history against that victim, yes. But, also known past MO.
However, devil’s argument, I do believe while “Cui bono” is near impossible to use in everyday cyber attacks, it is much more valid in nation state level attribution.
Typically, a reasonable, but largely ‘ad hoc’ analysis is made of the “level of sophistication required”, and then that deeply lowers the suspect pool.
But, there is another, even more important consideration to take note of in this:
What you hear about attribution from the authorities in clear counterintelligence cases in regards to methods and sources of attribution is very often likely to be at worst a lie, at best, significant details are left out.
That does not mean they may make pronouncements based on poor evidence, of course.
It also does not mean they may not make valid pronouncements.
They may feel very high confidence because of evidence it is “such and such country”, but not state this.
So, where they may have very strong evidence due to either technical or human intelligence, obviously, this is something they may want to hide.
Why do you never see them even stating this in articles or interviews?
If you tell some group you may have penetrated them, they may start investigations which could produce something. They very likely will increase their security, which is the last thing you want them to do.
If they are kept in doubt as to your confidence of detection, then their security will tend to be laxed.
Being able to cite more lean methods of evidence can operate as cover for more dependable methods of evidence gathering.
It is “like” law enforcement, but there are very strong and important differences.
Clive Robinson • March 11, 2016 9:49 PM
@ Mark H,
The “Cui bono?” (Who benefits?) heuristic is useful in identifying parties who may be motivated to take some action.
Whilst it does lead to a large number of potential suspects as a first approximation, the next step is to use what was once called “deductive reasoning” to reduce the number down. Though most of the time this appears to the public to lack reasoning and just be “follow the money” or some other formulaic response such as “previous form”. The reality in many places is “shake the tree” and “listen to chatter”, on the simple theory that criminals “have lose lips”.
However the problem with IC activities is they don’t usually have lose lips, or do things for base monetary reasons. Whilst they do often have form, it’s part of “imposed policy” and kept hidden. Which means with cyber-espionage more than half the time they are trying to make it look like somebody else in a false flag operation or “anybody and nobody”.
The times the ICs do become visable it’s to send a message to people. In the case of Russia energy manipulation for political control is known to be standatd policy. Likewise attacks against wealthy Russian businessmen who are felt to be straying etc etc.
Thus at first sight the whole attack appears overwhelmingly to be Russian for “Oh so many reasons”. But on second sight that also makes the Russian’s easy “fall guys” for false flag activities, which have been seen with respect to the Ukrainian issues. Which brings us onto “sending a message” which could be to or from Russia in equal measure, and there is insufficient information to say one way or the other.
Unfortunately what might otherwise be a “tell” of “no damage” does not work, all it really says is “state level”. Which under normal circumstances would point to somebody other than Russia on the idea of “Who has a history of assault against the victim” because Russia usually wants some “damage” to punch the message home. But the circumstances are not normal due to the involvment of Russian businessmen, and what gets fed back into the Russian economy from these energy businesses.
Interestingly it appears that those energy businesses also supply Russian areas of influence, so we can not say who the infiltraters are or what their original plans were because cutting power to Russian areas “in winter” would certainly send a powerful “two can play this game” message to Russia…
We have seen this “who’s the real target” problem befor with Stuxnet, the obvious country being attacked was Iran. But as I noted at the time, via Iran was about the only way to get at North Korea. Which was something that the US had as a much higher priority target at the time. People disagreed, but later the US did indicate that NK was what it was actually about for them…
As I noted above Russia has been making a nuisance of themselves in Syria which is upsetting many. For instance Turkey took direct action by shooting down a Russian plane. This would have acutely embarrassed the US, UK and others who for various reasons are apparently “talking not acting” thus appear weak to those on the ground opposed to the current regime. Giving Russian Businessmen close to Putin “cause for concern” is one of the economic tactics both the UK and US have used and are currently using to try to reign in Putin’s activities. Thus the attack certainly fits UK policy and the UK have considerable “previous” on cyber attacks.
So personally I would not rule out the UK or US for this attack. Because it has all the hallmarks of a firm message with “deniability” to all but the target, which again is WASP IC MO not Russian.
Buck • March 13, 2016 7:37 PM
@Some guy with experience
Based on your numbers of $200B/year, that is $625/ per citizen or $52/month/citizen. My gut says it isn’t quite that high, but let’s use it for estimates.
My gut says the same, but for naught other than a great thought-experiment, I’ll assume that this is a reasonable calculation for the moment…
There would be outrage if the industry adds a $1000 annually or 60% to your average bill
Yes. Although, would said outrage necessarily lead to meaningful change? This sort of government-mediated exchange is already in place thanks to the ACA. However, unlike the preexisting conditions of the financial/insurance markets, my proposal offers further benefits outside of our current sensibilities…
I don’t much care for the term ‘air-gap’ and while ‘energy-gap’ is a little better, it’s still a bit misleading. I really would prefer a ‘people-gap’ (which obviously comes with its own set of failures). The thing is though, personnel vulnerabilities are far more well-established and understood than the more obscure people+tech equations… Maybe we could try to constantly rotate 2+ partners in hopes of avoiding nefarious collusions..? (Double my ridiculous estimate again, is $400 billion / year really so outlandish now?) It’s tough to tease out the particulars of the size of subsidies for energy companies. This 2015 report from the U.S. EIA provides a clue:
Direct Federal Financial Interventions and Subsidies in Energy in Fiscal Year 2013
The total value of direct federal financial interventions and subsidies in energy markets decreased nearly 25% between FYs 2010 and 2013, declining from $38.0 billion to $29.3 billion
So, that is at least an order of magnitude lower than I’d need to make my point, but not all subsidies impacting the energy sector are included in this report… It certainly doesn’t include any DoD expenditures on grid security. It’s not likely that I would be able to figure out how large that number is. That may be for the best, assuming the grid is actually already secure enough against any potential threats!
And, about those other benefits I mentioned before?
Giving more people a larger stake in their own futures, having something to be proud of, higher-velocity cash-injections to keep capitalism running smoothly, social cohesion..? Do people still care about any of these things?
Subscribe to comments on this entry
Leave a comment
Sidebar photo of Bruce Schneier by Joe MacInnis.
K.S. • March 9, 2016 8:21 AM
Connecting crucial power grid infrastructure C&C to the internet should count as negligence and makes this hack less impressive.