Comments

Woody • May 8, 2007 4:59 PM

Having previous worked with SCADA systems, this is nowhere near the first vulnerability like this.

SCADA systems have been attached to modems for years (decades), with just a password to get in, and likely little monitoring of when someone dials in to the modem.

But the damage would have been limited in size to just that system, and I doubt would have been made public.

Ian Mason • May 8, 2007 5:23 PM

One major brand of phone exchange (as in central office or international gateway exchange) is controlled with a Windows 2000 PC sitting next to it. These are frequently installed on naked, high bandwidth internet connections in data centres by ‘phone people who don’t understand net stuff. Certainly the manufacturer’s engineers who installed one for me were completely un-savvy about the net and about locking down a Windows box.

These are sitting around the world, exploitable now. This is critical national infrastructure, albeit in private hands. In small to medium sized, cost sensitive telcos who run these they are typically bought as a turnkey solution and the technical staff running them day to day have none of the large telco procedures or experience to keep them secure and well maintained.

Timmy303 • May 8, 2007 6:18 PM

Didn’t Vitek Boden use protocol weaknesses to insert a fake station in a SCADA system and forge commands to valves and the like? Once these systems go wireless or connect to the PSTN, they’re open to all sorts of mischief. Perhaps a more central issue is whether or not SCADA implementations as an institution present a homogenous enough face to potential attackers that a single vulnerability would affect a large number of implementations. R. Clarke would have us believe that it does, but my experience has shown otherwise.

Dylan • May 8, 2007 6:42 PM

I might add that the Boden case was an “inside” job using equipment he had stolen from his employer, and that he was very familiar with. However, it was a remote job, wireless connection and a laptop from his car.

Paul • May 8, 2007 7:31 PM

This is just the start.

As SCADA network node developers convert to standard platforms (e.g. Windows) and protocols (e.g. TCP/IP) they will have to deal with coming up to speed with basic Internet hygiene requirements. Unfortunately they are coming from a world where systems were static (never needed a response plan to vulnerabilities) and hard to understand. Those assumptions are now patently false. One knee-jerk response is to disconnect all SCADA systems from all wide area networks, but as we’ve already learned in perimeter security, this can’t work in the long term.

Eventually the developers will have to come to the same realization the rest of us have – ALL systems need to be hardened to some degree against compromise and be ready to be updated when a vulnerability is found. You can’t predict who’s going to attack you, when, or why.

Gray Ghost • May 8, 2007 10:18 PM

This is definitely not the first major SCADA hole. However, it is one more.

It is incredible the lack of security found in many of these control systems as they are generally implemented and maintained by Controls Engineers with limited or no involvement of I.T. However, experienced control systems engineers that also have I.T. style security knowledge are incredibly rare.

The controls engineers are the right people (with help) to be building and maintaining control systems. I am just as concerned about killing somebody with a control systems safety failure as with a remote hack causing a loss of control. And yes, killing a person or 10 has happened due to a control systems safety failure.

Part of the reason experienced control systems security engineers are rare is that these control systems have been typically air-gapped. That trend started changing significantly in the late 90’s due to the increased engineering and management capability to have the read-only information available to users on the corporate networks. This now makes the networks connected – hopefully but not always securely.

Idaho National Laboratory (FERC/D.O.E. has demonstrated attack methods against SCADA systems at a SANS conference about last February in Orlando. SANS should continue this educational work.

Use of ancient operating systems is common. Many control systems still run WinNT today.

Scheduled downtimes, for example, of power plants frequently only happen annually. Extended downtime occcurs every 3-6 years. The software is not designed and usually cannot safely be patched while the plants are operating.

Digital Bond (No relationship) is one organization that specializes in SCADA security. They are working with major nulnerability assessment and intrusion detection products including Sourcefire/Snort and Tenable/Nessus. They also maintain an active blog.

Significant attention is being given to these issues by numerous organizations. Some examples include:

NIST SP800-82 Guide to SCADA and Industrial Control Systems Security

Instrument Society of America (ISA) SP99 – Manufacturing and Control Systems Security (early draft)

North American Electric Reliability Council (NERC) – Critical Infrastructure Protection Standards

SANS has also given the topic some attention working with several groups including MS-ISAC and D.O.E. regarding acquisition of secure control systems. I would recommend that the standard soon focus on maintenance and support contracts as well.

throx • May 8, 2007 10:37 PM

A properly implemented SCADA system has no connection to any sort of public network , or even a network that internal end users can connect regular Windows PCs to, and the control system engineers like to keep it that way. The far more critical concept is one of denial of service – if the user hits that big red button to stop the conveyer then you NEED it to stop within milliseconds, guaranteed, no matter what other things are going on over your network and so the network must have a deterministic traffic pattern.

In short, yes, better security is needed but as long as the security issues with the control network are well understood and access to that entire network is limited then it only becomes an issue of security-in-depth and not necessarily as doom-and-gloom as these companies are suggesting.

Polar bear • May 8, 2007 11:03 PM

If it turns out terrorists are able to exploit this, who do we get to blame? Iran?

Factory Auto • May 9, 2007 12:23 AM

The article is misleading, because it gives the impression that this is a problem for SCADA systems only. Any sort of factory automation system that uses OPC would have similar problems. Almost all the numerous industrial network protocols are proprietary, and OPC acts as a common driver interface between them and software running on MS Windows. Most of the OPC software vendors base their code off samples provided by the OPC Foundation (the specs are proprietary, like just about everything else in the automation industry).

Realistically, the only practical way any of these systems can be secured is to run them on their own network with a security “box” between them and the rest of the company. With all the industrial automation vendors moving their proprietary network protocols to Ethernet (Profinet, Ethernet/IP, EtherCat, Powerlink, etc.), trying to secure the plant level networks themselves is impossible. “Log-in and password” isn’t practical, anymore than it is for your mouse or speakers on your PC.

Tarkeel • May 9, 2007 2:17 AM

@Gray Ghost: You make it sound like the system can only be patched during downtime. A properly designed system with high-availability redundancy can be patched without taking it down. It’s not very common; in fact ClustRa is the only system I know of that does this.

egeltje • May 9, 2007 3:21 AM

@Woody: “But the damage would have been limited in size to just that system, and I doubt would have been made public”.
Last October, there was a controlled cut of a high-voltage line in Germany, which resulted in a powerloss for half Europe. Imagine if this single action was a not-controlled cut. One system, true, but the consequences are far, far greater.

@Trox: “A properly implemented SCADA system has no connection to any sort of public network”.
That used to be true 10 years ago. SCADA is more and more moving to a client-server model with centralized command (while keeping the real-time characterristics).

This real-time aspect is why a lot of security measures available on the market today are no good for SCADA. I cannot afford a virus scanner on my system, since I have to guarantee (by law) a certain response time.

Ian Miller • May 9, 2007 3:27 AM

Is it that the OPC protcols are flawed or is it the implementation of them that is flawed? There are implementations of OPC for platforms other than MS Windows, e.g. VMS.

I worked on SCADA systems many years ago and there where no MS Windows systems involved. Each part was custom built to be safe and robust. It is a concern that a platform that was not designed to be secure (MS Windows) is now being used for SCADA. When I worked on SCADA they where installed and the software then was not updated for years, and then only after very careful testing.

I can see the requirement to extract reporting information over networks and to use standard protocols but would hope that the fundamental attitude of safety first would still apply. However I expect the ‘race to the bottom’ affects the SCADA world as it does everything else.

David (Toronto) • May 9, 2007 5:47 AM

Definitely not the first case unless bad design doesn’t count.

About 10 years ago I evaluated the security of a SCADA system. The developer had reengineered the system from legacy to client server. Commands flew in clear across the wire. Care to guess where all the security logic was?

The solution:build a network fortress around a remote terminal solution using 2-factor authentication and VPN inside the WAN.

David (Toronto) • May 9, 2007 5:51 AM

What scares me just as much.

Consider all the medical equipment running windows systems hooked to networks.

The vendors typically certify the system at build time. Any modification is tampering and voids the certification. Patching is a modification.

Brings a whole new meaning to blue screen of death.

stephan beirer • May 9, 2007 6:19 AM

@egeltje

that control systems can not afford anti virus protection due to performance reasons is a kind of a industrial myth and might have been true 5-10 years ago. Of course I don’t know the special characteristics of your system but the NIST Special Publication 1058 might be of interest to you:

“Using Host-Based Antivirus Software
on Industrial Control Systems”

should be freely available on the NIST web site.

May I ask what kind of law or regulation (and in which country) demands which repsonse time and which can not be fullfilled when using AV software?

regards,

stephan

Gray Ghost • May 9, 2007 6:48 AM

@Tarkeel

Yes, the ability to patch is dependent on the system capabilities and the system design. While not generally a good idea for most SCADA systems that operate in high risk environments, systems can be very carefully patched while operating. I have done it many times and I have cleaned up enough messes where it has not been done properly.

Redundancy only buys part of the solution. While available, what percent of PLCs run redundant. Probably between 0.1 and 0.01%. At least these only are patched rarely. How about Ethernet networks with redundant routers/switches/firewalls and multiple paths? These are very easy yet still less than common.

greg • May 9, 2007 7:04 AM

@David

Life support systems, flight systems do not use windows for critical parts of there function. Its even in the EULA of windows and many other software products.

The critical parts are usally PLC or some other logic type device or microcontroller. The code audits are very very strict and time consuming making the software very very expensive even for a “simple” controller.

The windows box or other uncertified box is typicaly just used for monitoring. The life support systems I have worked on, the winodws cannot turn on/off or do anything. Only make the beep sound. The vast majory don’t even use “windows” for that either…

Gas SCADA Guy • May 9, 2007 7:35 AM

Sorry I’m late to this thread but I have a few comments as a system manager of various SCADA systems since 1979:

1) We only patch our systems after our SCADA software vendor has approved the patches and then we first install them on a test system and monitor before we roll out to production machines.

2) You can run AV software on SCADA workstations & servers and you can even use some of the existing corporate IT systems to help you do this but you have to be very careful with the implementation and you need to punch a few more holes in your firewall.

3) Our current system is a mix of Windows workstations & Unix servers but our new system will be all Windows based. I’m not really thrilled about that but it seems to be the way most SCADA vendors are moving as they don’t want to support multiple OS platforms.

4) SCADA security is a different animal from regular IT security in that reliability and ease of system operator use is paramount but I would think at least 80% of other “normal” IT security practices still apply to SCADA systems

5) There have been a FEW reported attacks on SCADA systems but most are either insider attacks or non-directed windows based ones that slip through firewalls.

6) IMHO, physical attacks on the various remote locations being controlled and telemetered into these SCADA systems are bigger security threats then attacks on the SCADA system itself.

Gas SCADA Guy • May 9, 2007 7:57 AM

“These backend protocols are often based upon standards that pre-date Windows,” Graham wrote in his blog. “They are horribly insecure because few people in the SCADA industry know what a ‘buffer-overflow’ is.”

Watch the FUD please as all SCADA system software developers know what a “buffer-overflow” is as do almost all SCADA support personnel. SCADA system operators and their managers may not know but why should they? It’s our job, SCADA support, to protect them from things of this nature.

Gas SCADA Guy • May 9, 2007 8:17 AM

“There have been a number of high profile incidents in the past few years: In Maroochy Shire, Queensland, Australia in 2000 a disgruntled ex-employee hacked into a water control system and flooded the grounds of a hotel and a nearby river with a million litres of sewage (see p25). In Russia, malicious hackers managed to take control of a gas pipeline run by Gazprom for around 24 hours in 1999. More recently, in 2003 the ‘Slammer’ worm, which caused major problems for IT systems generally around the world, disabled a safety monitoring system at Ohio’s Davis-Besse nuclear plant for nearly five hours.”

That AU Shire attack has become a joke amongst lots of SCADA pros as it’s brought up over and over, ad nauseum, in various articles and at SCADA security conferences but the facts are it happened in 2000 and it was an insider attack. Where are all the other SCADA attack reports since then?

I don’t think the Gazprom attack was ever confirmed even though it keeps popping up here and there.

The slammer worm attack is an example on a non-directed attack that can hit any unprotected Windows system so its relevance to SCADA is questionable except if you wish to argue that no SCADA system should be running on Windows, as lots of people do argue.

stacy • May 9, 2007 8:27 AM

Definitely not “the first remotely exploitable SCADA security vulnerability.”
Digital Bond http://www.digitalbond.com has been trying to engage the SCADA community in the “disclosure” debate for a while now; there is a lot of resistance to any such talk. They also have released a number of Nessus plugins for SCADA vulnerabilities.

Jeff • May 9, 2007 9:30 AM

I’ve been programming/managing “infrastructure” SCADA systems for about 20 years. Some general comments, most echoing what has already been written:

No, it’s not “the first remotely exploitable SCADA security vulnerability.”

If you only care about “terrorists” then yes, physical attacks are incredibly easier and would be much more spectacular than electronic attacks.

Back to computer systems, the insider threat is huge and the problem of ensuring that everyone is “cleared” in every part of the chain is nearly infeasible.

They use Windows because those of us who work with the alternatives are getting older and command higher salaries.

The industry regulations will do nothing the enhance true systems security. “Security Theater” at its finest.

Dan Linder • May 9, 2007 11:01 AM

@Greg

Too many things are now becoming critical to patient care that truly are running on unstable environments. Not just the common target of Windows, but any system made up of an OS, multiple databases, and “glue” software that presents the data to the health-care professional are re-defining the term “mission critical” to the civilian world.

A co-worker was at a demo given by a health-care automation company. It consisted of a theatrical play by the staff showing a hypothetical patient going in for a visit with his doctor and some mysterious pain (presumed to be a heart blockage). The whole premise was that their system would allow the family doctor to update this magical “cloud” of data that the surgeon in the operating room could reference for some highly critical observation he had made in the initial exam. The take-away thought was “Without their technology, the surgeon would have made a critical error during surgery and the patient would have died.”

Unfortunately, in the live demo that my co-worker attended, this wondrous system locked up and that “life saving data” that the physician diligently entered into his tablet PC didn’t get into the master database and the over-head display of the surgeons console just displayed a blank page.

I cringe every time I hear about a hospital being on the “cutting edge” of technology and wonder how many of their tools are just held together with bailing wire and good luck…

Dan

AJ • May 9, 2007 12:45 PM

There was a relevant talk on the subject of PLC security at a conference I attended a couple of weeks ago, on some work done at CERN. The talk is accessible on the http://ftp.desy.de FTP server, at /pub/EPICS/meeting-2007/Control%20Systems%20Under%20Attack%20@%20EPICS%20(2007).pdf
The full URL is ftp://ftp.desy.de/pub/EPICS/meeting-2007/Control%20Systems%20Under%20Attack%20@%20EPICS%20(2007).pdf but I’m not sure that it will pass through this comment system unscathed.

Jay • May 10, 2007 11:23 AM

SCADA-type systems will be widely deployed and connected to the Internet as “intelligent grid” systems. This is inclusive of electric meters which will integrate connect/disconnect switches and extends to other systems that will control loads on a wide scale and will connect to the electric utility’s own networks, and, given a couple of hops, to the Internet.

In 3-5 years (hopefully a bit longer) threats like this will become well-known as blackouts and other system “anomalies” become more common. On the security front, we will take a couple of steps back as we try and move forward.

Bryan Singer • May 10, 2007 6:56 PM

Having worked in SCADA and controls security now since 2001, and IT security back to 1991, I think I can safely say that this is absolutely not an understated threat. The risk to critical infrastructure from domestic or international remote exploit, be it incidental or malicious, is quite high. What I think is misleading is that in many installations we are only a small percentage of the installed base of systems, and in the past these were not networked as they are today. Further, most control systems have limited logging and alarm features when it comes to security incidents. As such, its hard to put a true number on the actual incidents.

Looking at some of the sources of data, however, we know that we have quite a challenge on our hands. Eric Byres database (byressecurity.com), has tracked just over 100 incidents recorded (which were fully documented with permission, making this data set small but likely of high value). Personally from my customer base, I am aware of about 300 cases, most of whom will never be documented (most operators and companies don’t like to advertise, “I’ve been hacked!”)

This is certainly not the first remotely exploitable condition, I can think of about 10 off the top of my head without working hard at it against several major vendors. I have seen ways to exploit MODBUS/TCP, CIP, Ethernet/IP, and just about every other protocol used in these systems out there. These are networked and routable protocols, making them vulnerable to exploit (as we all know, every protocol has its vulnerabilities).

While this sounds a bit like “chicken little,” the concern here is that the risk exposure associated with SCADA and controls is exponentially larger. We have immediate risk to life and limb at the facility, and the chance for more protracted issues during distribution. Losing email or a webserver has public consequences, but doesn’t under normal course threaten lives.

Of course, this is not just an IT problem. Most companies have huge physical and personnel security issues, and do very little for incident management, security awareness training, etc. I have never, not once, had a system that we could not compromise when asked to do so. We come at them sideways, sometimes, but an intentioned person would make short work of bringing down most critical infrastructure. I feel pretty confident in saying that any of us that have been working in this space for any time probably have the knowledge required to stop a significant amount of manufacturing, disable infrastructure, stop utilities, turn off the lights, water, etc without a lot of effort. If we know how to do it, so do the proverbial “bad guys” (or they shortly will).

Just look at the past two weeks, two events made the news:

Chlorine theft in water utilities in SoCal, excessive sodium hydroxide (lye) causes chemical burns to people in Spencer, MA (I don’t know what caused the release, but it likely was a significant amount). Having looked at a number of our utilities, I can say that it can often be scary just how much could be done. For a few of us out here, it is a personal mission to identify and mitigate issues as quickly as possible.

While not necessarily remotely exploitable conditions , it does demonstrate that awareness is on the rise, and people will want to go after these systems eventually. We hoped it would be 3-5 years when we started ISA-99 in 2001/2….. I would say we aren’t seeing as many incidents as some of us thought, but we are seeing enough in my mind to justify the time and expenditure we are putting into this space. I just wish it were properly directed….

Jacob Brodsky, PE • May 10, 2007 8:43 PM

As a SCADA system engineer (and integrator, programmer and chief bottle washer) let me be the first to assure you that Bryan Singer is on the mark. I’m going to go one step further, Mr. Schneier:

We know all about security, but we call it by a different name: Safety. Security is but one aspect of a much larger issue that is largely alien to IT folk: The system must stay alive and online while the process is running. The notion of rebooting while a process is running keeps most of us up at nights. We SCADA engineers have insisted on a level of performance for decades that IT has only recently discovered with high availability servers on the Internet.

Furthermore, live patching is not what this is about. Our systems must behave in a consistent manner (ever heard of ISO 9000, FDA 22CFR11? –you know, the regulations and standards from years ago that actually set the marks for designing such systems?) You all know about blown patches. Do you want to have to explain to a town that the chemical burns they received in their drinking water was due to a malfunctioning SCADA alarm system?

Mr Schneier, we don’t patch like the IT folks do. We TEST all patches first to be sure that they do not conflict with ongoing concerns in our systems. We don’t just throw security features at the wall. Read that draft NIST SP800-82 document if you want to know why security isn’t handled why you think it ought to be.

Live patching indeed! I’ll patch live when you can guarantee that there is no chance a patch will malfunction, leak memory or break one of many links in a complex system. We validate everything Mr. Schneier. We won’t take the word of a software firm that this or that patch is good. I’ve seen far too many blown patches to ever take someone’s word for it.

Albatross • May 13, 2007 3:26 PM

My experience with SCADA systems is that the entire network is ridiculously vulnerable, protected largedly by security-by-obscurity. For example:

1) The vendors with whom we dealt refused to allow their products to be assessed, or even port-scanned. Their claim was that doing so might crash the software (a claim about which I suspect I need make no comment here).

2) The SCADA appliance ran on a completely unremediated default Solaris OS that was years old. Based on a moment of research on the Solaris/Apache version of the SCADA appliance, we typed “GET \etc\passwd” into port 80 on a telnet connection and were handed the un-shadowed password file to crack at our leisure.

3) The arrogant senior network administrator insisted that the LAN and SCADA networks were completely separate. When asked why his desktop Windows NT 4 system (this was not long ago) had two network cards, he revealed that one went to the LAN, and the other to the SCADA network. When asked if this didn’t violate the “completely separate” design, he looked at our auditor as if he were stupide and pointed out that his NT system had a routing table that kept the two networks separate. Again, no comment necessary.

4) While physical security at the data center was robust, the seventy electrical relay stations distributed near the US/Mexico border that we examined had less-than-no security. Most stations had sensors to detect door opening, but also had air conditioning units mounted in the walls. These units could be pushed into the building allowing access. Once inside the trespasser would find technical manuals and schematics to help invade the SCADA network, including unchanged default vendor passwords.

5) After delivering our report on these and many other security vulnerabilities, the client manually whitewashed the report to reflect no security vulnerabilities, and offered that report to their Board.

Rick • October 2, 2007 7:25 AM

Hmmmm, Ironic?

I noticed that the CNN clip was preceded by a Blackberry advertisement, and a Blackberry ad then appeared next to the video frame, at least when I viewed it.

A Blackberry instead of a mouse click to trigger an attack?!? 😉

Viz • December 4, 2007 1:40 PM

I was a controls engineer who is now a software engineer. In the mid 90’s I worked for a company that did pneumatic tube systems in hospitals, banks and even Home Depot.

We did 2 types of systems, one was an OS/2 system written in rexx, the other was a completely imported tube system that ran on it’s own embedded software.

The OS/2 system was remotely accessible via modem. The other you needed to be onsite to operate.

The OS/2 system communicated using a current loop and hex codes that told the tube routers, gates, and stations what to do.

Guess the password on that modem and you could easily wreck the system. However there were manual procedures in the hospitals that handled a system outage.

Since the OS/2 boxes weren’t connected to the hospital LAN they weren’t an attack vector for the rest of the network but you could cause havoc if you got into one.

The worst it would cause is a bunch of orderlies to do a lot of manual running around, or you could slam a carrier and break the contents necessitating a massive clean up of the system.

I’m not saying all SCADA is like this, but this is one example of what’s running out there…

-Viz

Citizen • January 16, 2008 7:49 AM

Vitek Boden the man jailed for the Scada attack on the waste plant in QLD Australia got out of jail change is name to Peter Markan and got a live in job on a Island resort called Couren Cove Resort as a technician ,soon started his tricks again causing faults on the resort power station Scada and blaming another fellow worker that just started working for the resort until was found out,worried of being fired and arrested once again follow the worker to a remote area on the Island and try to kill him,the police charged Vitek Boden(Peter Markan) Jully 2007 with assault causing bodly harm.This Guy is a lunatic and did not learn by his time in jail,just change his name, lied about work references and carry on hacking with disregard for safety of other human beings .Case going to court in 2008.For more details just ask

rod • January 21, 2008 3:52 PM

Hey Citizen,

Do you know if he is currently being held in custody? and if so where?

Where can i obtain some more details of what he did in CC? and where and when the court case will go ahead?

Thanks in advance

sillybuffer • November 22, 2008 12:53 AM

Hello to all,

Has anyone heard the results of the Vitek Boden/ Peter Markan trial I belive it has taken place?
Is he doing time in the “big house”?

Anoni • October 7, 2009 6:05 AM

Vitec Boden /Peter Markan was convicted at Southport Court and jailed for the period of four years on the last week of November 2008 and I have no dough this twisted guy will do again once he get’s out just add a plc/scada system

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.