Attack against Florida Water Treatment Facility

A water treatment plant in Oldsmar, Florida, was attacked last Friday. The attacker took control of one of the systems, and increased the amount of sodium hydroxide—that’s lye—by a factor of 100. This could have been fatal to people living downstream, if an alert operator hadn’t noticed the change and reversed it.

We don’t know who is behind this attack. Despite its similarities to a Russian attack of a Ukrainian power plant in 2015, my bet is that it’s a disgruntled insider: either a current or former employee. It just doesn’t make sense for Russia to be behind this.

ArsTechnica is reporting on the poor cybersecurity at the plant:

The Florida water treatment facility whose computer system experienced a potentially hazardous computer breach last week used an unsupported version of Windows with no firewall and shared the same TeamViewer password among its employees, government officials have reported.

Brian Krebs points out that the fact that we know about this attack is what’s rare:

Spend a few minutes searching Twitter, Reddit or any number of other social media sites and you’ll find countless examples of researchers posting proof of being able to access so-called “human-machine interfaces”—basically web pages designed to interact remotely with various complex systems, such as those that monitor and/or control things like power, water, sewage and manufacturing plants.

And yet, there have been precious few known incidents of malicious hackers abusing this access to disrupt these complex systems. That is, until this past Monday, when Florida county sheriff Bob Gualtieri held a remarkably clear-headed and fact-filled news conference about an attempt to poison the water supply of Oldsmar, a town of around 15,000 not far from Tampa.

Tags: , ,

Posted on February 12, 2021 at 6:08 AM • 23 Comments

Comments

water engineer • February 12, 2021 8:06 AM

Poor cybersecurity at water and wastewater plants is real thing. But this was an event with an extremely low potential of any harm.

I’m skeptical that the NaOH could be dosed at such a high amount to actually harm someone downstream. Regardless, if an attacker was really interested in harm, they’d shut off the disinfection chemicals and let the bugs in the water do the work at the faucet. But in reality, out-of-range pH and Cl residuals on the pipe leaving the plant likely would have sounded alarms.

Karlo • February 12, 2021 8:17 AM

Hey, Lithuania is one of the three Baltic States – Latvia, Lithuania and Estonia. Lithuanian culture is closely related to Latvian, and their languages share a number of features and common words.

Craig Finseth • February 12, 2021 8:20 AM

Not to mention that increasing the injected amount by a factor of 100 would cause the supply to run out quickly (100 times faster…) and the large amount would be diluted by the total volume of water passing through.

Serious, but probably not fatal to anyone in the city.

A_Water_Industry_Guy • February 12, 2021 8:45 AM

There were many things being done wrong or poorly here. People will reasonably differ about whether some practices should be allowed ex. internet access to water plant networks and equipment.

On the specific point being sensationalized: in no water plant that I have been in (mostly industrial, but some drinking water) would it be physically possible to increase the injection rate of sodium hydroxide over 100x. Just because the screen says so, does not make it so.

Pumps are sized at some factor of maximum system flow – say, twice. There might be a backup pump, so now you have maximum of 4x. At minimum system flow it might be, at the outside, 10x. Maybe unpleasant to drink but still not dangerous. If it injecting into a holding tank while nothing is being pumped out, to a dangerous point within the tank, there are still safeguards downstream to prevent release of water out of specification.

Water plants won’t pay for pumps or tanks larger than needed to do the job.

I would be surprised if the Oldsmar plant was designed, maintained and operated very differently.

There’s no guarantee, of course, that every drinking water plant in the world meets a minimum standard. You also can’t 100% guard against an inside job, whether through malice, hubris or stupidity (I guess the last two are intertwined). Do an internet search on Walkeron water for one example.

Despite the sensationalism that doesn’t help – increasing people’s anxiety, and directing attention away from the many problems in drinking water infrastructure and operation that do need to be fixed – this incident getting press worldwide might finally put enough attention on the problems for fixes to start getting made. One can hope.

Jon • February 12, 2021 8:56 AM

These kinds of stories are rare because the first rule of industrial infosec is “Don’t talk about industrial infosec.” It’s not necessarily because these kinds of incidents are rare.

MB • February 12, 2021 9:10 AM

I would echo the skepticism of water engineer. These systems have acceptable ranges, I would be surprised if it could have made it out. My guess is an alarm sounded (either NaOH or pH out of range) and alerted the operator. One aspect about this. Irán used these systems in uranium enrichment plant, and these were exploited there. It HMI/SCADA system that uses PLCs to operate/monitor the process.

wiredog • February 12, 2021 9:24 AM

A major problem with these systems is that while properly securing it when it’s installed is easy and inexpensive, keeping it secure over time is difficult and costs money. At the least, with an externally connected system, you have to validate every software and hardware change to that system from a security perspective. Which probably means having a security engineer type available. You also have to update your systems to mitigate new threats. Running Windows7 after it’s EOLed is a security risk, but replacing it with Windows 10 costs money, and what if the hardware won’t run Windows 10? And on, and on.

When I did industrial automation a couple of decades ago the primary security was to not have it plugged into a network unless it was being remotely serviced.

Uhu • February 12, 2021 9:28 AM

I would bet it probably was an employee changing the values by mistake (fatfinger). But they don’t dare admitting that, and the plant doesn’t have the means to figure out who it was (since they all share the same password). As a face-saving measure they now claim they were hacked.

MB • February 12, 2021 9:40 AM

One thing that seems to be missed, when I worked on these systems (water and wastewater) there was always an “air gap”, I.e. they were not connected to the internet. Running old software is not surprising (cost and if it works why break it?). I would be shocked if this was not an inside job, given the old software and the general vendor recommendation not to put these systems on the internet.

me • February 12, 2021 9:46 AM

an attacker increase the value by 100 times the normal value
Them: DON’T WORRY! it’s not that the change is immediate and it’s phyisically impossible for the level to go that high (probably because there is not enough supply)…

this doesn’t make me feel any safe…
it’s like if someone hack a heating system and set it to 1000 °C it doesn’t really matter that the heating system can go at max 30 °C and not 1000. what matter is that someone have full control over it and that the hack have been discovered BY BEING LUCKY

JonKnowsNothing • February 12, 2021 10:17 AM

@All

There are many incidents at such facilities but they often are either dealt with In House or only rise to the level of Local Public Interest. Large MSM media rarely report on these small town incidents unless something spectacular happens.

There are computer, manual control, pumps and gates that can fail. Physical plant failures such as percolation ponds/dams collapsing. These happen regularly and are ho-hum to most media.

What is an Open Secret, occasionally trotted out by MSM as a “Oh My!” story, is that much of the infrastructure of the USA is crumbling. Localities have neither the money nor the expertise much less the political will to deal with city utility problems like sewer, water, etc. Anything that extends beyond their borders is guaranteed to Not Happen Soon.

The leaking pipes of all kinds, have led some cities to ban Gas Piping to new houses because of huge gas explosions due to aging leaking gas lines. Water systems everywhere deal with not just purification but the increasing amounts of Forever Toxic Chemicals in the water. Sewage systems fail regularly in some districts leaving pools of raw effluent flowing out of pipes with failed sewage pumps requiring HazMat Teams to “Scoop The Poop” and contaminated soils.

There are so many attack surfaces physically, that even though a swimming pool of raw sewage will get noticed, the response isn’t always to Clean It Up.

It all depends on where the sewage comes from.

If it’s human within the City Borders it will likely get Cleaned (you might need to replace the pool yourself). If it’s human and has been openly dumped into rivers, streams and the ocean under “Legal Release” laws, you get to surf in the poo. Dog runs on the beach are nice but hardly anyone removes the dog waste which rolls out on the high tide and comes in on the low tide. If a hog farm holding pond dam fails (1), no one is going to clean it up. In communities with “financial, social and demographic” issues, often Nothing Happens and the locals live with the results.

These problems are not new news. They are replicated in nearly every infrastructure system. There are no incentives to fix it and there are no incentives to replace it.

Then you have the Flint Michigan Water Problem (2) where thousands of people were forced to use toxic water piped into their homes by the city, because a neoliberal wanted to get more money from them. This sort of computer failure is not from a hacker setting a value “out of bounds”, but from purposeful treatment settings. No “override” or “alarm” is going to go off because the system has been deliberately setup to deliver the toxic product.

The computer security is nil, as are the physical plants.

1a, Earth dams and settlement ponds fail regularly. Often squirrel or burrowing animals dig into the dam sides weakening the walls. Heavy rains and local flooding can over top or erode the sides. Vast hog waste ponds dot the country. The rate of hog production far exceeds the ability to dispose of the waste through natural methods and no hog farmer is going to build a huge hog sewage plant. Cattle feedlots have similar problems as do Dairy production farms. The industrial size of these facilities are based on maintaining the lowest cost possible. A bulldozer is cheaper.

1b, Agricultural Chemicals percolating into the water table like Dibromochloropropane (DBCP) (ng/L). A now banned nematocide that may still
be present in soils due to runoff/leaching from former use on soybeans, cotton, vineyards, tomatoes, and tree fruit

ht tps://en.wikipedia.org/wiki/1,2-Dibromo-3-chloropropane

2, ht tps://en.wikipedia.org/wiki/Flint_water_crisis

JR • February 12, 2021 10:18 AM

@all

The problem in the USA is that the regulated private sector and the Government are complying with different cyber regulations. It needs to be standardized.

There’s 16 critical infrastructure private sectors, 10 of which are regulated by DHS CISA. But CISA doesn’t have enforcement capability. Krebs just told Congress yesterday that they should expand CISA to the other 7 sectors (except DoD??) and give them enforcement capability. The problem with this is that Treasury which examines banks is one of the sectors. GAO even recommended the same thing in a September, 2020 report. However, the 4th Amendment of the US Constitution has to do with warrants and protected data. To give DHS CISA warrantless oversight of banks, it has to first strip out Federal Leo from their department in order for CISA to enforce cybersecurity across the private sector. IMHO I think the better solution is for their to be new Federal Cybersecurity Department that is autonomous and not under DHS. It could have dotted line to DHS and DoD which need to be in sync on this too. Autonomy is required by NIST anyway. Even in the private sector cybersecurity needs to be independent. There’s so few people in the world that are experts in this field it makes no sense that each industry sector has its own cyber regulations and no means to inspect or enforce it. It needs to be standardized.

Vendors selling to Government or even State Government need to comply with FISMA which is NIST 800-53. But all they do is promise to comply when they sign the sales contract. That is entirely ridiculous. Here the Government is pushing “Zero Trust” cybersecurity design, yet there’s no means to certify their vendors. Vendors selling to the critical infrastructure private sector should be assessed and approved by a non-governmental body so there’s no means for pay to play.

What has occurred in FL is very common. If we had a nationwide Cybersecurity standard, we could then make EOL (end-of-life) systems illegal and add criminal charges to those companies and persons who violate these laws. Some regulators have started assigning criminal charges to cyber violations. But the US Gov also needs to publicly identify and bar vendors and contractors from selling to regulated sectors who repeatedly are involved in these breaches.

The ICS and SCADA vendors who sell to the Water Treatment plants, the DoD, DHS, banks and pharma are the same. But they have different cyber regulations to comply with for each sector. That is insanity. There aren’t enough cyber experts in the world to achieve this so vendors hire people who have no cyber experience to feign the appearance of compliance instead of just complying and they pay more for that too, in more ways than one.

SOX needs to be updated for Cybersecurity and the CMMC assessment needs to be required in every GSA contract plus added to FISMA contract language.

The EPA manages the cybersecurity for Water treatment plants.

Krebs testimony to Congress 2 days ago. Link broken.
https://www. fedscoop.com/krebs-congress-cisa-qsmo/

Impossibly Stupid • February 12, 2021 10:35 AM

This sort of thing is somewhere in the “a vulnerability is not an exploit” and “security through obscurity” space. There may not be any reason to get scared over this particular incident, but it does serve as a good wake up call for considering what other things could be done in the “Click Here to Kill Everybody” future. I think the biggest problem was the shared password, leaving no accountability, which means the management must be blamed.

xcv • February 12, 2021 11:07 AM

The attacker took control of one of the systems, and increased the amount of sodium hydroxide — that’s lye — by a factor of 100. This could have been fatal to people living downstream, if an alert operator hadn’t noticed the change and reversed it.

You know the far-right conspiracy theories about “the Jews” poisoning the wells.

I can’t say that it’s even primarily “Jews” rather than mostly self-righteous Christians who are responsible for such abominations in most U.S. municipalities, but it’s downtown, and there’s a “district” with a “community” water supply.

Why does a system like that even exist? What’s wrong with private wells out on the farm? Or, say, a system of collecting rainwater from the roof and filtering it for private use?

There are hookers downtown, women with a water well agenda: eww, gross, there are bugs in it, we have to make sure all the water is clean and sanitized and fluoridated.

xcv • February 12, 2021 11:38 AM

A person can be “reasonable” and “non-chalant” about all this, but it really happened in Germany and Poland in the 1920s.

The Germans have always had a culture of really strict laws, and a lot of churches, much ado about official membership rolls at each church, if someone isn’t welcome at a church, then maybe he/she is Jewish, if the local Jewish congregation is willing to accept a new member, as for example in the Book of Ruth.

But to expel someone from church and to make such a sharp distinction from the Christian churches, is suspect.

Do Jews believe in Christ? Sure, that’s the Greek name for the Messiah, the Anointed One, that’s all well documented in the Scripture.

But to accept Jesus as the Christ goes against the religion, customs, and practices of most Jews.

And I don’t know that Jesus freaks are taking care of or properly handling — or refraining from tampering with — wells and water supplies any better than the Jews of industrialist Germany.

And people have a lot of secret religions and hidden superstitions that center around wells and witching for water. There’s always a Satanic agenda of “population control” with false ideals of herd immunity, vaccinations, fluoridation, routine extraction of wisdom teeth and so on and so forth, where not even a normal childbirth can be handled except as a grave medical emergency, with the usual C-section of the mother, circumcision of the child etc., which all fits into the devilish obsession with “clean water” on the part of government.

Etienne • February 12, 2021 11:43 AM

The opportunity for security professionals is out there. There should be no way to connect from outside to infrastructure designed to sustain life, and prevent death. Such as water, power, sewer treatment, or even traffic signals, and radio networks.

It seems inconceivable to me, that these outside connections do not require professional level hardware authentication and encryption.

I’m retired from the defense industry, where the Open Internet was used to tunnel secure networks (even Top Secret). In no case was anything connected directly to remote or base level secure sites. They all had hardware NSA approved interfaces to encrypt and authenticate.

Cities and counties should have these same devices.

Our switches and routers, networks and VLAN’s were all sophisticated enough to detect unauthorized physical attachment, above the normal network attachment (usually 100% fiber).

How can any utility or city infrastructure be allowed to operate and be licensed for operation, with anything less?

Anytime something like this occurs, the utility should immediately have its license revoked, and steps taken within hours to recertify operation after it has been corrected, or shut down permanently if they can’t meet the standard.

I too think this was an inside job, or leaked intelligence from someone working inside. They may have made copies of manuals and documented operations that were themselves copied and used by acquaintances.

Usually a person isn’t so inclined to throw their job away, but their crack smoking anarchist child may be reaching the age of joining the 27 club.

David Leppik • February 12, 2021 11:48 AM

Having these systems computerized and/or online makes them more vulnerable, but at the end of the day we can’t protect all soft targets all the time. Expecting every small-town water engineer to keep up with the latest security protocols is too much to ask, particularly when there are so many other soft targets waiting to be exploited.

Keeping software up-to-date is a good idea, so long as it doesn’t break anything (a real risk in these older systems that were designed for install-and-forget.) Keeping it off the Internet is also good for security, but not guaranteed safe, as Iran discovered with its centrifuges. But there’s no guarantee that that’s a good trade-off either, since it may be better to have an expert monitor dozens of them remotely.

Security experts need to have a certain amount of paranoia, but at the end of the day sometimes the only way to know what’s a real risk is to wait to see what gets exploited. In this case, it’s not clear that a nefarious state-level actor could actually be more dangerous than incompetent management—which is something that human nature requires water systems to be robust against.

Some Guy • February 12, 2021 3:10 PM

One item often missed is this is a small town brackish water reverse osmosis plant. Small as in 5000 water meters, 1.5 million gallons per day, the plant manager is the lead operator, and a total staff for the plant of 7 (per the cities budget book online). The capital budget includes a SCADA upgrade for $42K. The operating budget is $1.5M. This isn’t Tampa, their next door neighbor, with 30 times the population and far more industry.

In this environment, there are not backup operators, maintenance crews, engineers, and others when things go wrong. You call another operator and get help. The manager checks in on things when he can, but he doesn’t live there.

Likely, the lead operator has never even heard the phrase multi factor authentication. Or if he has, that sounds complex. So you call the IT team with their team of 4 and their only major accomplishment listed on the city website was installing Wifi at the city office and, by chance, the water plant.

Before you laugh and ask why they do this, it isn’t that unusual (other than reverse osmosis). It is real world. There are many small town water systems and electric coops where the IT person is also the manager, the toilet cleaner, the regulatory compliance person, and answers the phone (which is listed on the town’s website).

The better questions to ask are:
– Is this the right model? (In rural areas, it may be the only model, but here, not so sure)
– what is the right way to get the skills and resources to protect critical infrastructure facilities without raising costs so high that people have to go back in time to the 1800’s?
– Since regulations really matter in a municipality, what are the right regulations to impose that can be complied with? There are 50+ state/territorial regulations as there is no inter-state impact.

JonKnowsNothing • February 12, 2021 10:51 PM

@Some Guy @All

re: Small Water Plant for Small Towns

Too many urban dwellers presume that What They Have is What Others Have.

It causes no end of problems when urban dwellers move to rural areas and get The Big Surprise that this is Not So. Of course, urban dwellers exercise their Elite Rights and demand to have exactly what big towns and cities with big budgets can provide them.

One small town I know of, there are ZERO on site Anyones. Weekly a contract person drives around the water ponds and settlement basins doing a Drive-By-Visual inspection for squirrel damage. There is a phone number to call M-F if you notice a problem. If there is a Big Problem you call the fire department.

It’s not that the plant is fully automated, it’s that there isn’t any money in the city budget to hire the positions. Not that many folks want to live in small rural areas where there’s not a latte on every corner, or mega shopping malls and the big deal is the annual 4H/FAA livestock event and the perfume of the day is Eau De Veau.

ht tps://www.washingtonpost.com/nation/2019/09/06/maurice-rooster-france-lawsuit/

ht tps://www.theguardian.com/world/2020/jun/19/maurice-the-noisey-french-rooster-dies-aged-six

ht tps://www.independent.co.uk/news/world/europe/ducks-quacking-court-case-dax-france-dominique-douthe-a9209386.html

ht tps://www.theguardian.com/world/2021/jan/21/france-passes-sensory-heritage-law-after-plight-of-maurice-the-noisy-rooster
(url fractured to prevent autorun)

Clive Robinson • February 13, 2021 2:36 AM

@ JonKnowsNothing,

Of course, urban dwellers exercise their Elite Rights and demand to have exactly what big towns and cities with big budgets can provide them.

Or can not provide them…

I lived in an urban area however where gardens were not small. So I used to keep ducks and chickes for both eggs and meat and likewise geese, rabbits and goats[0] from time to time. I used to pickup the majority of their feed as the cut offs/outer leaves at the local out door fruit and veg market. The stall holders were glad to give it to me because they otherwise had to pay to have it taken away.

Now back then my neighbours were all people who had lived through WWII and understood what a garden and an allotment could give you in the way of food and were happy to recieve eggs etc as gifts.

Yes just like humans, animals make noise and smell, it’s all part of life’s great tapestry though it does not feel like it on a friday night at pub/bar kicking out time where human noise and smell is positively worse in oh so many ways.

But times change and younger families moved in as older ones moved on. All of a sudden your “cute livestock” their children liked to watch and cared not a jot about sound nor smell about becomes not so nice… Worse the do not like the idea that “Jemima Puddle-Duck”[1], Nanny Goat, Peter the Rabbit etc and other storybook creatures their children so love are going to end up on the dinner table etc. The children never appeared to mind when they were under ten and it was only those just comming into their teens that fussed (and realy they should have been reading something a bit more intellect challenging than story books).

I sometimes wonder how the children of such “coseted children” come to terms with where a beefburger, or hamburger, sausage, in a bun or even baccon for a sarnnie come from. And by that I do not mean cow/beef, pig/pork, or sheep/mutton[2] but what part of the animal thus why “thick end”, “thin end”, “Chop”, and all the other interesting terms like “T-Bone” come about[3].

Maybe they don’t maybe they assume the meat leads a dull uninteresting and above all quiet life growing sedately in a white polystyrene tray[4]…

[0] Goats provided they are not some pedigree breed are fairly easy to keep and if you “grow’m right” provide milk, cheese and meat. The meat usually tasts more like people think lamb should taste than sheep actually do. So go better in most dishes that require mutton rather better than what passes for mutton these days. Especially if you like curry or other strongly spiced dishes such as those from the middle east.

[1] https://en.wikipedia.org/wiki/The_Tale_of_Jemima_Puddle-Duck

[2] Ever notice how what we call it when alive is “old english” but when on the table “old french”? Well there’s a long story behind that 😉

[3] Also why some are castrated to keep certain joints of meat so tender. Oh and what the testes got called before being put on the table (don’t confuse your sweetmeats and sweetbreads).

[4] OK, I accept the fact that turkeys would be that dull and incapable of existence, if not for the fact they have the brains and outlook of your average “Friday night pugalist when fuled up on corn mash distilate”.

Frank Wilhoit • February 13, 2021 3:54 AM

@wiredog,

Many things like this come down to accounting rules. The initial acquisition and installation of a system is a capital expenditure. Maintenance is an operating expenditure. (So is training.) The capital and operating budgets are allocated and governed entirely differently. The operating budget is very often compromised in order to meet overall bottom-line targets for a quarter or fiscal year.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.