Schneier on Security

Clive Robinson • November 21, 2011 8:40 AM

Whilst I agree with @ Vince Mulhollan it looks like poor engineering practice it raises an interesting problem.

I personally after many years have not seen an “accident”, I’ve generaly only seen lack of forsight and bad risk analysis.

However I’ve also seen having worked in petro chem and similar that you cannot defend against every individual risk even if you can foresee it, there just is not the technology or the money to do it (as the US realised with NORAD’s mountain location and the Russian 50Mton T’sar Bomba).

So how do you deal / mitigate the problems.

My take is to actually not defend agaist individual risks unless very high, but defend against classes of effect for other foreseeable risks and finally go for a generalised distributed architecture to try to defend against the unknown.

Whilst the latter strategy works for tangable real world risks and attacks, it fails miserably against intangable information world risks and attacks, simply because every thing is effectivly at the same point of time and space for all practical purposes.

So how do we actually defend against information effects caused by poor design and or attack?

Clive Robinson • November 21, 2011 11:20 AM

Anybody else notice the similarity between this and the RSA embarrassment a while ago?

That is the attack was made possible by a third party storing authentication credentials on a server for “technical support” reasons but also connecting it to a network which could be accessed from outside the organisation by unautherised persons…

I wonder how many other third party organisations do this and how long it will be before they wake up and secure such valuable databases in a suitably secure way?

Clive Robinson • November 22, 2011 1:00 AM

With regards @ Dirk Praet’s comment on risk,

“By factoring in their real weight when making our business impact analysis.”

That it should be an on going process with a good “weather eye” for the future storms blowing in.

Sadly the opposite is usually true in that “business Impact analysis” tends to be done at some point in time for any given project then given “a stamp of approval” and then filed in an archive box somewhere and not get updated.

Worse they often get copied without modification or update from one project to another. Oh and nobody ever reads them untill something goes wrong anyway (anyone remember the story doing the rounds of the “Deepwater horizon” “environmental impact assessment” sent to the EPA for approval that supposadly talked about the arctic?)

The important thing is “at a point in time”, prior to Stuxnet for well over a decade I was one of the lonely voices in the wild saying it was stupid to connect SCADA systems to the Internet or any other outside access point. Some people in the industry would politly (pretend) to listen and then ignore the advice because it stood in the way of “competitive business drivers” and “there was no proof there was a problem”…

I would just as politely tell them there was proof and talk about what had happened in the telecommunications industry where PABX’s and even CO switches were getting regularly owned. The response was usually something along the lines of “different industry” or “that’s to get free phone calls, no such incentive here”.

Even when certain people in the US started crowing about blowing up a Russian energy instalation in the worlds biggest non nuclear fireball and droping hints that it was “an agency action” in general the responses were the same (mind you I’m on record as saying I thought the claims were a crock, and why).

Then there was that (supposed) demonstration by a US agency of how software could be used to destroy a generating set.

And still few if any were listening, the UK water and energy producers appeared to be the exception, mainly due to “still paying lip service” to the clauses put in the deregulation by Maggie Thatcher’s “civil uprising” and “ash city” paranoia in the 1980’s which saw a rebuilding of underground bunkers etc (remember being paranoid does not make you wrong only obsesive 😉

Now even after Stuxnet in most places it’s the same arguments as always, so many are not waking up and smelling the coffee even though it’s scalding the hands of their industry friends.

And to be honest, I can see why…

In an unregulated cost sensitive market the infrastructure is the biggest capital cost, and where annual maintenance costs are often as large as profits it can be a case of take a risk or price yourself out of business.

Unregulated markets are almost always a race for the bottom unless the price of technological inovation makes it cheaper to upgrade than not (which has been the case with software control over hardware control).

It has been seen in both Canada and New Zeland, what happened in their unregulated power markets was base infrustructure was neither correctly maintained nor upgraded on the excuse of “if it ain’t broken we don’t need to fix it” from the directors providing “shareholder value” on a quater by quater basis. Eventually their luck ran out (as it does for everyone) and they basicaly bleeted “not our fault” before disappearing behind a wall of lawyers to let other pick up the pieces and importantly the cost (I’ve mentioned this before on this blog when talking about cascade faliures).

Which is why security of infrastructure should be mandated by regulation to prevent a “race for the bottom” which can only result in infrastructure failure, which always hurts badly.

Now the 64trillion dollar question is what happens now…

Remember infrastructure generaly has a quater century or more of return on investment time. What is going to be the cost of bringing everything up to spec and over what time period, and how much damage is going to happen in that intervening time window (which could easily be ten to thirty years).

Oh and one of the big lessons from Stuxnet, is “air gapping” is nolonger a valid temporary or permanent fix, we realy have to get in as close to the metal as possible and fix security there.

With “air gap” crossing all attacks are “insider attacks”, “perimeter defence” is only going to keep out the script kiddes for a year or two untill they get given “air gap crossing attacks”.

It’s sad realy because I worked out how to do the various tricks to cross air gaps and mentioned them on this blog long before Stuxnet reared it’s ugly little head. I talked about them specificaly as a way to get at “voting machines” via the maintanence staff with “fire and forget” malware. It makes me wonder if the attackers listen more than the defenders…

So remember when you hear people talking about “firewalls” and “SSH” and all those other security solutions that have been around for twenty years and failing us, just remember that’s “perimeter defence” and it’s now without any kind of doubt a “known fail”.

With “air gap” crossing attacks, there is no “perimeter”, no “choke point” every host is effectivly connected to the internet via somebodies memory stick or maintanence man with a software upgrade.

Likewise don’t think “code signing” is a solution, it’s not, again I was warning just how much of a fail it could be and mentioning on this blog why long before Stuxnet proved it to be the case.

So in the short term get out your old books on “Bastion Hosts” and remember every system that is connected to your network is a host that can be owned. If you cannot harden it get out the wire cutters and cut not the network connection but the power cable. Otherwise it could be your organisation we will be discussing here.

I’ll leave it to Nick P and RobertT to give you the further bad news about OS security and Chip security.

Clive Robinson • November 22, 2011 4:28 PM

@ suckaforagoodstory,

“Maybe the point is to leave holes, so we can take it down when we want and if these open our own systems to control from folks who have access then these are the breaks??”

You would hope not, whilst the best for of defense is offence, it is only true if your rear is well protected which means you still need strong defences irrespective of your offensive capability.

However, your comment can be likened to a small pile of kindling and twigs to which you have successfully struck a spark into a small flame.

Please alow me to add a gallon or two of very high octaine fuel to make it a merry blaze.

In the US Industrial Control Systems (ICS) have their very own Computer Emergancy Response Team known as ISC-CERT run by non other than the Department of Hapless Security.

A few weeks back they pulled a master stroke by reclasifing their problems away…

I will let you read the version of events posted by Ralph Langner who did so much to tame stuxnet, he to put it mildly sounds gob smacked,

http://www.langner.com/en/2011/09/23/dhs%e2%80%98-new-semantic-approach-to-risk-mitigation/

Yes Mr DHS Martty Edwards has decided that over 90% of security faults with ICS software are not ICS-CERT’s reponsability to inform people of… But you should still tell ICS-CERT of everything you discover so they can pass it on to “the favourd few” who the DHS have gathered together…

Thankfully unlike others commentators on this Ralph has not resorted to unseamly personal comments about Mr Edwards.

Clive Robinson • November 23, 2011 6:37 AM

@ David

Thanks for posting that, it actually made me laugh for a number of reasons one being,

There is no evidence to support claims made in the initial Fusion Center report – which was based on raw, unconfirmed data and subsequently leaked to the media

That the Fusion Center has a bigger “leak” than the water plant.

Speaking of leaks the DHS (ICS-CERT) communication does remind me of the story of “The little Dutch Boy” plugging a hole with his finger.

It is such a strongly worded denial made before they have compleated their investigation, such things are almost never made by engineers only politicians because they have a habit of comming back to haunt you.

For instance if the water company is connected to the Internet, and they are keeping logs of inbound TCP/UDP/etc connections as most firewalls can, I would be very surprised if they did not see connections from Russia, China and a whole host of other places, most people see such connection attempts on a daily if not minute by minute basis in their logs. Analysing such data can be a difficult task at best with an uncertain outcome. As an over generalised case you can only realy rule it out when you have a directly attributable cause that clearly shows no involvment of Internet traffic.

For instance the result is apparently a burnt out pump motor, this can happen for a whole host of reasons including manufacturing defects, wear and tear, poor maintanence, operator error and quite a few others. In most of these cases it is actually difficult to find “proof positive” of the cause just an assumption. The reason is there is rarely sufficient instrumentation to tell and most times the instruments that are there are not sufficiently well logged, or their sampling window is too great to give sufficient detail.

Now we know from “The Auora Attack” it is realisticaly possible to get “inside control loops” and cause problems specifically in that case by opening and closing relays between a generator and a power network. When a generator under load is taken off load by the relay opening it’s speed increases and a phase shift between it’s ouput and that of the power network occurs. On closing the relay an EMF pull back occurs. This puts a sudden undew stress on the generator shaft and the turbin driving it. Generaly the control loops are sufficiently damped to prevent feed back effects giving incorrect control signals due to various response delays resulting in positive feedback and thus some form of oscillatory behaviour (often called “loop hunting”). The damping also prevents alarm conditions when the generator is “normaly” connected or disconnected from the network. The Auror Attack uses the loop damping to hide it’s activities, specifically it opens and closes the relays in a very short period of time providing very short sharp torque shocks that don’t get through the instrumentation damping to raise an alarm.

The result is the generator or the turbine driving it dies a very early death, and it would ordinarily be attributed to some unknow defect in manufacturing, maintanence etc or if insufficient logging is performed operator error is assumed.This is the same as with air and sea accident investigations, if a pilot / captain is not alive to tell their side of the story then the aircraft / ship was lost due to “pilot error” / negligence.

Now the thing is as I’ve said on a number of occassions transducers work both ways, that is a microphone can act as a speaker and a speaker can act as a microphone. It is the same with motors and generators. The speed of a motor and importantly the current it draws from the supply is controled by it’s load and the back EMF generated. Importantly it is the back EMF of a motor that stops it drawing a very very large current from the supply (the DC resistance of a winding is very very small for efficiency reasons).

Now when you open the relay between the motor an the supply a number of things happen, firstly load inertia keeps it turning, secondly the motor becomes a generator thirdly if it is an AC motor it slips with regards to the supply phase.

Rapidly opening and closing a power supply relay will cause not just stress on the actuall pump shaft but also the motor windings and depending on the supply, that as well.

If the relay opening and closing is done in the right way, the other instrumentation feeding back to the local control system loop will not see it and therefore neither will the SCADA system which is the only place any instrument logging is likley to be done.

Now I’d better make one thing clear these days when a power control engineer talks about a “relay” they are not talking about the old fashioned mechanical device you might find in your thirty year old boiler. They are generaly talking about a solid state device that is quite complex and driven by a microprocessor. Because there is a microprocessor in there modern relays tend to be very complex and are quite often “network hosts” in their own right or are connected to a small network host. As with any other network host they can be “hacked” and ephemeral rouge code can be added to their RAM (thus not leaving any rouge code behind after the relay is powered down to repair the pump etc).

Thus they are just as vulnerable as the motor drive controlers on the centrifuges that Stuxnet targeted.

Clive Robinson • November 23, 2011 2:17 PM

@ Bruce,

CSO Online has an article related to this,

http://www.csoonline.com/article/694707/experts-advise-caution-information-sharing-in-wake-of-alleged-utility-attacks

One thing I strongly object to in it is,

MANDIANT’s Bejtlich argues that while mandatory reporting to a central critica infrastructure CIRT (Critical Incident Response Team) would be”a step in the right direction,” he doesn’t agree that public reporting will do much to improve critical infrastructure risk posture. “Public breach reporting isn’t necessarily going to improve security within critical infrastructure. Since 2006 I’ve advocated creation of a National Digital Security Board to investigate important incidents. NDSB reports do not need to”name names” in order to have a positive impact on security.

This is tantamount to the “favored few” mechanism whereby the Government (usually for political reasons) only lets some people know about security vulnerabilities, thus leaving many many others vulnerable.

At the end of the day it is in everybodies interest to have all vulnerable systems fixed promptly. The reason is as with Stuxnet, you realy cannot control where malware ends up or where it decides to go next, thus vulnerable systems just act as infection agents/vectors.

And the idea of “stacking up” vulnerabilities to use as “cyber-weapons” frankly I find absolutly pathetic, it shows that those involved have little idea about such things. As I’ve noted before,

Whilst ofense may be the best form of defense, it requires that your home base be secure otherwise you will find your self wining a battle but losing the war.

Clive Robinson • November 24, 2011 4:55 AM

@ RobertT,

“I don’t really understand your objection.”

As with many. things (with me 😉 it’s a bit complicated.

As you note once a specific attack vector is used it loses it’s value very quickly. However the loss of value is directly related to the number of vulnerable systems and the proportion that remain unpatched etc.

As we know from past experiance the speed a manufacture/supplier provides a patch is very much related to how many customers know about it.

We already know of atleast one major ICS software and systems supplier that has a number of critical vulnerabilities, that they have decided to ignore and thus are selling know to be critical vulnerable systems.

The fact that ICS-CERT’s Marty Edwards has basicaly decided not to make the vulnerabilities concerned a public reporting function of ICS-CERT (but the “favourd few” get told) means that the major supplier is under no real preasure to mend it’s broken systems.

I fully expect this ridiculous state of affairs to get considerably worse and for the likes of ICS-CERT to effectivly become pariahs as the industry realises they are being hung out to dry by the overly politicized idiots.

Now one thing that is known that malware that exploits vulnerabilities has a habit of turning up in unexpected places. Stuxnet was supposadly highly directed but poped up in all sorts of unintended places. Stuxnet was supposadly written by “experts”, so if the supposed experts get it badly wrong what does it say about other malware written by people not so expert, which would be the majority of malware writters.

Thus any cyber-weapon written by the US to be used against other countries is likley to come back at US organisations using ICS who are not part of the “favourd few” as well as US allies and totally uninvolved bystanders. Some of these non target nations may well suffer major harm and thus will regard the US malware as a serious attack on their Sovereignty, which for those who don’t know is “a decleration of war by deed” and against many international treaties.

Now a Sovereign nation could respond in many ways some by direct action against US persons within their borders, some by direct economic action of seizing all US assets, and some by retaliating in kind, which will lead to rapid escalation which is not good for anybody, potentialy the economic effects would be worse than a limited scale nuclear war.

The problem for the US is it is probably the number one “Internet dependent” nation in the world so the US gets it worse than any other nation…

Put simply for the US it is more in their interest to ensure all ICS systems are secure than for any other nation, this appears to be a point lost on the “war hawks”.

Then there is the question of who comprises the “favourd few” and why, I sincerly doubt that anybody in the US DHS has actually sat down and worked out what the “true critical infrastructure” is. Modern manufacturing and society are so intermeshed it is difficult to determine exactly what would happen if an ICS cyber-weapon touched down in the US. For instance we know the “utility” organisations are considered critical, but what about “food producers”, “medical supply producers”, “iron and steel producers”, etc, etc.

What do you thing would happen if tommorow 30% of the processed food producers could not produce their goods?

Depending on who you beleive something like half of US households do not know how to cook beyond the very very basics, certainly not enough to be able to make bread or pastry or cook meat safely. This is radicaly different to just a generation or so ago, and is largely due to the rapid increase in “single member house holds” in the “proffesional classes and the asy availability of “ready-meals”. Oh and don’t say they could go to a restaurant, the major chains (ie fast food outlets) are all criticaly dependent on ICS to industrialy produce food to the point where it can be “cooked in store”. Then there’s the canning and botteling plants…

So rather than,

If this secret process results in information asymmetry, whereby certain countries / individuals benefit, than I’m guessing I’ll benefit more often than I loose, so No problem

The more industrialised the nation the worse off it will be and the number one vulnerable nation is the US currently, so not “No problem” but “Big problem”.

Oh and “certain countries” like China can bring the US to near economic colapse just by being a bit awkward over shipping certain goods in a timely manner. And they can do this at any time just by deciding to “improve customs checks”.

And we know China are well aware of this, and are acting in a way to make the situation worse for the US and other western nations, just look at what is going on with the likes of rare earth metals which is one of their more blatent tactics…