Nasty Cisco Attack

Cisco Systems officials are warning customers of a series of attacks that completely hijack critical networking gear by swapping out the valid ROMMON firmware image with one that’s been maliciously altered.

The attackers use valid administrator credentials, an indication the attacks are being carried out either by insiders or people who have otherwise managed to get hold of the highly sensitive passwords required to update and make changes to the Cisco hardware. Short for ROM Monitor, ROMMON is the means for booting Cisco’s IOS operating system. Administrators use it to perform a variety of configuration tasks, including recovering lost passwords, downloading software, or in some cases running the router itself.

There’s no indication of who is doing these attacks, but it’s exactly the sort of thing you’d expect out of a government attacker. Regardless of which government initially discovered this, assume that they’re all exploiting it by now—and will continue to do so until it’s fixed.

Tags: Cisco, hacking, malware

Posted on August 19, 2015 at 12:34 PM • 30 Comments

Comments

Max • August 19, 2015 1:06 PM

The routers were probably delivered with the implant already installed.

The manufacturers need to provide a procedure, not dependent on software, for verifying the firmware.

Anura • August 19, 2015 1:15 PM

@Max

It’s possible, although that the advisory says “In all cases seen by Cisco, attackers accessed the devices using valid administrative credentials” which makes me suspect that there are some logs showing it happening after they are up and running. You don’t need physical access to the equipment to upload new firmware – it could simply be someone with malware installed on an administrator’s machine. Without more information when it’s a little hard to form conclusions..

rob • August 19, 2015 1:40 PM

I am puzzled.

“In all cases seen by Cisco, attackers accessed the devices using valid administrative credentials”

How would you observe a corrupt ROMMON if you didn’t happen to notice the logs of someone installing it? How is anyone noticing this at all? Other than command logging or looking for a spontaneous reboot AND someone paying attention.

Wait.

System Bootstrap, Version 11.1(10) [dschwart 10], RELEASE SOFTWARE (fc1)
Copyright (c) 1994 by cisco Systems, Inc.
C7200 processor with 65536 Kbytes of main memory

CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Self decompressing the image : ###########################################################################

WELCOME TO EVIL ROMMON LOL. GUESS I SHOULDN’T HAVE SAID THAT – SORT OF BLOWS THE JOKE.

rommon 1 >

More to the point: any local compromise of IOS getting to priv level 15 would allow this to happen un-logged. That’s a serious problem. Cisco should be signing their images and should be measuring them to a third party – somehow.

Douglas Gourlay • August 19, 2015 1:42 PM

Bruce, this is not surprising at all. Lets break it down to a few points:

1) This is what happens when vendors fail to signature sign software, BIOS, ROMMON, etc. there are not a lot of people in the world who know Cisco IOS/NX-OS well enough to write a corrupted firmware build, but I’ve worked with many of them at Cisco, Arista, Skyport – etc. It is do-able. This is like a BIOS level undetectable rootkit on every switch, router, firewall made by Cisco.

2) Couple the fact it is possible to be written with poor practices on using bastion hosts/jump servers to segment the management system and it is not difficult to have a user get access to a switch/router management interface.

3) Or worse this fictitious belief that ‘network virtualization will segment my admin network and people put the management network physically in-band, but logically separated by a protocol like VXLAN. News flash: VXLAN does not do any source MAC/IP/VNI verification – you can spoof your way past the jump server in about 12 seconds and a few lines of Python.

4) The management/administrative control model is still very weak. Once you ‘enable’ I can turn off logging, load a new ROMMON, bounce the box, and turn logging back on and most traces of my activity are gone.

5) IOS has no support for 2FA…

I don’t know why people hack servers – hack the switches and routers… then launch into anything you want…

Lar • August 19, 2015 5:03 PM

Oh Cisco, Oh Pancho!

LessThanObvious • August 19, 2015 5:30 PM

@Bruce – Until what is fixed? If someone is able to obtain your admin credentials and get any access to your equipment it’s pretty much a given that they own your network. Do you think there is something we aren’t being told about how credentials are obtained?

Per Arstechnica:
“The significance of the advisory isn’t that the initial firmware can be replaced. As indicated, that’s a standard feature not only with Cisco gear but just about any computing device. What’s important is that attackers are somehow managing to obtain the administrative credentials required to make unauthorized changes that take control of the networking gear. The advisory doesn’t say how the attackers are obtaining the credentials.” by Dan Goodin, Attackers are hijacking critical networking gear from Cisco, company warns.

albert • August 19, 2015 5:34 PM

@Lar,
¡HAHAHAHAHA!
. .. . .. O

albert • August 19, 2015 5:56 PM

@Douglas,
According to Cisco, “… To use the ROM monitor, you must be using a terminal or PC that is connected to the router over the console port. …” Please don’t tell me you can do this over the network.
.
Wouldn’t that require physical access to the router? Seems fairly secure, if you can trust your own people. Except for NSA interdiction enroute to the customer, of course.
.
How about ROM? Oh, and be sure to epoxy it real good. I can see the NSA contractors trying to unsolder ROM chips.
.
. .. . .. o

Router_Jockey • August 19, 2015 6:32 PM

@albert

Lots of Cisco routers have a dial-up modem attached to the console port so they can be managed remotely. It is “just like being there”. Welcome (back) to war dialing!

Tony H. • August 19, 2015 6:39 PM

@Douglas Gourlay
“5) IOS has no support for 2FA… ”

Um… You can set it up to use a RADIUS server for admin authentication, and that RADIUS server can be configured to require a hardware token or other 2FA mechanism.

Thoth • August 19, 2015 7:27 PM

The smartcard and TPM industry have done a pretty good job at ensuring security of their system which Cisco might want to look into. A tamper resistant chipset with signed hashes and signing keys inside the keystore of a tamper resistant chipset and a special command to check for tampering would turn into a much harder obstacle.

Dirk Praet • August 19, 2015 7:51 PM

@ Router_Jockey, @albert

Lots of Cisco routers have a dial-up modem attached to the console port so they can be managed remotely. It is “just like being there”. Welcome (back) to war dialing!

That’s really old school. Most network admins I know use a console server like Opengear’s ACM5000 Remote Site Manager.

The point, however, is that whoever is behind this found a way past the admin credentials (assuming the routers in question were properly secured and had passwords set for both in-line and console access). Which does beg the question: were these obtained through social engineering, network sniffing, a keylogger, or is there a hidden “service account” on these boxes?

Al • August 19, 2015 10:07 PM

the accusation smells fishy itself. first of all, targeting ROMMON is pointless unless there had been a code previously injected into the chip set. and by previously I mean when the device was being made in Cisco factory or anywhere that it’s being made. It is a fact that user password in Cisco devices are weak and can be easily decrypted, but attacker needs to have the hashed value for that in the first place to do that. but after that there is the Enable secret that every Cisco admin in his right mind would use it and this time the password in not reversible. it uses an one way MD5 hashing mechanism. and we all know that if we want to make any changes on a Cisco device, we need to have the enable password as well.

if we want to make an ideal environment of networking, I’d recommend using TACACS+ instead of RADIUS. RADIUS uses UDP and doesn’t encrypt the sessions that are made. radius only encrypts the transactions for passwords. but TACACS encrypts the whole session.

Key management issues would make it a nontrivial task to switch over to a stronger reversible algorithm, such as DES. Although it would be easy to modify Cisco IOS to use DES to encrypt passwords, there would be no security advantage in doing so if all Cisco IOS systems used the same DES key. If different keys were used by different systems, an administrative burden would be introduced for all Cisco IOS network administrators, and portability of configuration files between systems would be damaged. Customer demand for stronger reversible password encryption has been small.

(Ref: http://www.cisco.com/c/en/us/support/docs/security-vpn/remote-authentication-dial-user-service-radius/107614-64.html)

me • August 20, 2015 12:38 AM

Would there be any way to detect that this was happening?

whaddayaknow • August 20, 2015 1:18 AM

Kudos to Cisco for coming forward with this.

Does anyone doubt that other vendors’ equipment is just as vulnerable?

me • August 20, 2015 2:12 AM

I meant, are there signs that an end-user could be on the alert for, that would be likely to appear if a router were compromised.

Coyne Tibbets • August 20, 2015 2:47 AM

Wow. Who could have predicted this would happen with Cisco products? What hint has there ever been of Cisco products being exploited?

Oh. Right. JETPLOW.

Sancho_P • August 20, 2015 4:13 AM

@Dirk Praet wrote:
“whoever is behind this found a way past the admin credentials”

Hacking a Cisco router? NOBUS could do that.
Um, probably someone found the golden key provided for National Security?

No, wait.
Snowden was working at NSA.
Remember Syria 2012?
Snowden is against us.
Putin captured Snowden.
Putin is against us.
Snowden needs money.
… Snowden leaked golden key to Putin.
Putin needs money.
Putin sold the info to China.

==> Now we have evidence the “government attacker” is China.

Simon Josefsson • August 20, 2015 8:50 AM

I find that the ability of attackers to replace the firmware with a malicious firmware interesting — that indicates that attackers either reverse engineered the Cisco firmware and added malicious code, or that the attackers had access to Cisco source code and build environment for the firmware code. Both hypothesis are alarming.

/Simon

Evan Rowley • August 20, 2015 8:58 AM

Hopefully the newer Cisco NX-OS has relatively more robust ways of ensuring integrity, confidentiality, and non-repudiation of the firmware images. Does anyone on here have first-hand experience with the Cisco Nexus/NX-OS product line? Can you speak at all on how it compares to IOS in this domain?

Evan Rowley • August 20, 2015 9:01 AM

IMHO I think regular network admins were tricked somehow into uploading malicious firmware.

Evan Rowley • August 20, 2015 9:14 AM

To me it seems most probable that some network admins somewhere were MITM’d while downloading the new firmware to be used for their regular network maintenance. I think that would be the simplest attack vector for this. That scenario would be consistent with Cisco’s explanation, but phrasing it that way would direct more blame on to Cisco.

The best thing a Cisco customer could do would be to ensure that any command/control traffic between the compromised devices and outside world was restricted or at least monitored. Also ensure that similar firmware for security control devices such as firewalls, application layer gateways, etc. – were all controlled with a higher degree of security.

Clive Robinson • August 20, 2015 9:26 AM

@ Simon Josefsson,

I find that the ability of attackers to replace the firmware with a malicious firmware interesting

That is true of pretty much all Internet connectable hardware PCs and Interface cards these days.

The reason is that the use of SoC’s with embedded Flash ROM and Flash ROM for BIOS code is the manufacturing standard. The reason for this is not just the reduction in manufacturing cost, but also the reduction in rework and thus returns costs.

When the IoT and Smart Meters get going they will all folow this path and like implantable medical electronics, I don’t expect the security to be very high, and hidden pasword backdoors the norm.

In otherwords “seventh heaven” for the snoopers, malware writers and security services, and “hell on earth” for the rest of us.

As for getting source code and tool chains, those doing Reverse Engineering and the more sophisticated malware do not need anything beyond the ROM contents and modern debugging tools. Producing altered or new code whilst not trivial is well within many peoples capabilities. The hard part in some cases is getting the “code signed” or past the checker if it exists. Many if not most people that use code signing do not take anywhere near the number of precautions they should to protect the process…

Look at it this way, what background checks do you think most companies do on regular employees let alone “summer of code” / unpaid interns / work experiance students?

L Skeptic • August 20, 2015 9:28 AM

Did the affected companies have their router passwords in plain text on network shares or in their email archives? The Chinese are exfiltrating so much data that they are having to schedule their exfiltrations from US companies. If these companies had router passwords in the clear in emails, etc as companies often do that could explain the use of known passwords.

tinky winky • August 20, 2015 10:24 AM

@LessThanObvious

Disabling remote updates would solve this.

CCIE RS • August 20, 2015 12:41 PM

@Douglas Gourlay

TACACS allows authentication, authorization, and accounting controls as granular as you want to make it. Different privilege authorization levels for different users can be set centrally, independent of needing or knowing an enable password. Accounting will collect every command a user enters.

5) IOS has no support for 2FA..

Again, TACACS allows authentication out to AD, LDAP, RSA token, Kerberos might still be an option, can’t remember.

I seriously doubt any network that has backdoored ROMMON images running in it had their routers compromised as the entry point. These things go from spearfishing, to endpoint compromise, to local admin, to lateral movement inside, to domain credentials, to everything. Those routers were probably compromised via legit credentials because there were key loggers on the network engineers end workstations.

As far as doing IR against this kind of compromise, I’m not sure where to start. All network management systems will track IOS code version, I don’t recall any that also track ROMMON version. Detecting this in a network might be tricky.

LHarrison • August 20, 2015 8:18 PM

@Al

the accusation smells fishy itself. first of all, targeting ROMMON is pointless unless there had been a code previously injected into the chip set

why is that? it sounds like they are logging in through some backdoor (either one created by Cisco for their own use, or one created for NSA, like JETPLOW) and after that install their own ROMMON to ensure the kind of connectivity they desire.

Anyway this is pretty bad news for a lot of companies that rely on Cisco hardware.

vgor • August 20, 2015 8:38 PM

Look at it this way, what background checks do you think most companies do on regular employees let alone “summer of code” / unpaid interns / work experiance students?

Its easy to blame the defunct apprentice who has no defense, but what is really at stake here? Cisco’s reach makes it a prime intel target, so is there any reason not to look into the supply chain? Most if our gears were manufactured, and sometimes designed by, on foreign soils.and the relative ease of nation state exfiltration is quite unnerving these days.not sure if Mr trumps closing the borders is such a good idea.

John • August 21, 2015 5:35 AM

There are enough controllers and other vectors on various IOS platforms. ROMMON just happens to be a very easy to see component/high bang for your buck (also quite programmable from having pwned privilege level 15 credentials). Consider the number of devices with FPD images… These are programming FPGAs and CANBUS controllers and so on! There are several busses in the boxes (example, some platforms use i2c for this or that and is fairly externally accessible) that can be used for attack. Hell, considering some of the processors used in the boxes, it would probably be reasonable for a sophisticated adversary to buy a pwned drop-in replacement without much thought. All of these internals are publicly documented, albeit in a (sometimes) scattered way (not that it needs to be for the type of adversary likely involved).

All the signing in this world doesn’t keep a good adversary from putting something on a PCI bus and having full DMA access (no, the customers wouldn’t notice even if they deployed them all the time). “State security services” are not the main adversary planned for by equipment vendors due to the difficulty of actually isolating the various components that can be used for such attacks.

I think that the old adage that the one with physical access owns the box rings true again. Based on prior “revelations” concerning the activities of certain state security services, it is clear that the supply chain has to be addressed.

Twirrim • August 22, 2015 8:53 AM

People have been asking Cisco to require firmwares to be signed for a long time. I remember network engineers venting frustration at this some 10 years ago. It’s just astounding that they still haven’t implemented simple security steps, and have just relied on the (arrogant?) belief that malicious parties would be unable to create firmware for their devices.

Nasty Cisco Attack

Comments

Leave a comment Cancel reply