Router Vulnerability and the VPNFilter Botnet

On May 25, the FBI asked us all to reboot our routers. The story behind this request is one of sophisticated malware and unsophisticated home-network security, and it’s a harbinger of the sorts of pervasive threats ­ from nation-states, criminals and hackers ­ that we should expect in coming years.

VPNFilter is a sophisticated piece of malware that infects mostly older home and small-office routers made by Linksys, MikroTik, Netgear, QNAP and TP-Link. (For a list of specific models, click here.) It’s an impressive piece of work. It can eavesdrop on traffic passing through the router ­ specifically, log-in credentials and SCADA traffic, which is a networking protocol that controls power plants, chemical plants and industrial systems ­ attack other targets on the Internet and destructively “kill” its infected device. It is one of a very few pieces of malware that can survive a reboot, even though that’s what the FBI has requested. It has a number of other capabilities, and it can be remotely updated to provide still others. More than 500,000 routers in at least 54 countries have been infected since 2016.

Because of the malware’s sophistication, VPNFilter is believed to be the work of a government. The FBI suggested the Russian government was involved for two circumstantial reasons. One, a piece of the code is identical to one found in another piece of malware, called BlackEnergy, that was used in the December 2015 attack against Ukraine’s power grid. Russia is believed to be behind that attack. And two, the majority of those 500,000 infections are in Ukraine and controlled by a separate command-and-control server. There might also be classified evidence, as an FBI affidavit in this matter identifies the group behind VPNFilter as Sofacy, also known as APT28 and Fancy Bear. That’s the group behind a long list of attacks, including the 2016 hack of the Democratic National Committee.

Two companies, Cisco and Symantec, seem to have been working with the FBI during the past two years to track this malware as it infected ever more routers. The infection mechanism isn’t known, but we believe it targets known vulnerabilities in these older routers. Pretty much no one patches their routers, so the vulnerabilities have remained, even if they were fixed in new models from the same manufacturers.

On May 30, the FBI seized control of toknowall.com, a critical VPNFilter command-and-control server. This is called “sinkholing,” and serves to disrupt a critical part of this system. When infected routers contact toknowall.com, they will no longer be contacting a server owned by the malware’s creators; instead, they’ll be contacting a server owned by the FBI. This doesn’t entirely neutralize the malware, though. It will stay on the infected routers through reboot, and the underlying vulnerabilities remain, making the routers susceptible to reinfection with a variant controlled by a different server.

If you want to make sure your router is no longer infected, you need to do more than reboot it, the FBI’s warning notwithstanding. You need to reset the router to its factory settings. That means you need to reconfigure it for your network, which can be a pain if you’re not sophisticated in these matters. If you want to make sure your router cannot be reinfected, you need to update the firmware with any security patches from the manufacturer. This is harder to do and may strain your technical capabilities, though it’s ridiculous that routers don’t automatically download and install firmware updates on their own. Some of these models probably do not even have security patches available. Honestly, the best thing to do if you have one of the vulnerable models is to throw it away and get a new one. (Your ISP will probably send you a new one free if you claim that it’s not working properly. And you should have a new one, because if your current one is on the list, it’s at least 10 years old.)

So if it won’t clear out the malware, why is the FBI asking us to reboot our routers? It’s mostly just to get a sense of how bad the problem is. The FBI now controls toknowall.com. When an infected router gets rebooted, it connects to that server to get fully reinfected, and when it does, the FBI will know. Rebooting will give it a better idea of how many devices out there are infected.

Should you do it? It can’t hurt.

Internet of Things malware isn’t new. The 2016 Mirai botnet, for example, created by a lone hacker and not a government, targeted vulnerabilities in Internet-connected digital video recorders and webcams. Other malware has targeted Internet-connected thermostats. Lots of malware targets home routers. These devices are particularly vulnerable because they are often designed by ad hoc teams without a lot of security expertise, stay around in networks far longer than our computers and phones, and have no easy way to patch them.

It wouldn’t be surprising if the Russians targeted routers to build a network of infected computers for follow-on cyber operations. I’m sure many governments are doing the same. As long as we allow these insecure devices on the Internet ­ and short of security regulations, there’s no way to stop them ­ we’re going to be vulnerable to this kind of malware.

And next time, the command-and-control server won’t be so easy to disrupt.

This essay previously appeared in the Washington Post

EDITED TO ADD: The malware is more capable than we previously thought.

Tags: , , , , , , ,

Posted on June 11, 2018 at 6:19 AM • 39 Comments

Comments

tyson • June 11, 2018 7:10 AM

For people that have a vulnerable model, perhaps loading 3rd party code such as OpenWrt/LEDE would be a better option, rather than throwing it out and filling up our landfills and oceans with yet more plastic..

Vesselin Bontchev • June 11, 2018 7:50 AM

One, a piece of the code is identical to one found in another piece of malware, called BlackEnergy,

No, Bruce, it’s not “identical”. The similarly with BlackEnergy is that both have implementations of the RC4 cipher that are faulty in one and the same way. And while this, by itself, is suspicious, this flaw has been known since 2009 and just about anyone could have included it on purpose.

Of course, I’m not privy to any classified information regarding this issue, but the above, plus the fact that many of the compromised servers are in Ukraine seems pretty slim to me as attribution evidence. Maybe it was indeed Russia whodunit, but it’s not the publicly available evidence that shows it.

This doesn’t entirely neutralize the malware, though. It will stay on the infected routers through reboot, and the underlying vulnerabilities remain, making the routers susceptible to reinfection with a variant controlled by a different server.

“It” is a bit misleading here. The malware is modular and highly complex. The reboot will remove a part of the malware which is capable of bricking the device – which is why it is a very good idea to do it. It will not remove the part of the malware that talks to the C&C server.

If you want to make sure your router cannot be reinfected, you need to update the firmware with any security patches from the manufacturer.

You can’t know that. Nobody knows how exactly the routers are infected. The malware found on them does not contain an infection module. Maybe it’s via some exploit – but we don’t know which one and we don’t know if the latest firmware version is free from the corresponding vulnerability.

And you should have a new one, because if your current one is on the list, it’s at least 10 years old.

That’s not always a good idea. Newer devices tend to be “smarter” – which means “more vulnerable” (google “Hypponen’s Law”). One of the routers I have is on the list of vulnerable brands – but it’s not a vulnerable model, because it’s too old. If I update the firmware of another one, the new version will disable a critical capability, which capability I need for proper implementation of our VPN.

So if it won’t clear out the malware, why is the FBI asking us to reboot our routers? It’s mostly just to get a sense of how bad the problem is.

Also in order to disable the self-destructing capability, as explained above.

me • June 11, 2018 8:40 AM

@schneier
though it’s ridiculous that routers don’t automatically download and install firmware updates on their own

Be careful on what you ask! they might do it, and your router one year later might say “i’m not anymore supported so i’m going to brick myself, if you want to go in internet buy a new one”.
i DON’T want that my router or any other thing update automatically.

also: they will do some dumb mistake so that people can abuse the auto update feature and push malware using the update process (see for example “evilgrade”, a software designed to do exactly that)

me • June 11, 2018 8:50 AM

there is also a problem with updates:
-i have found a vulnerability in a router:ssh was enabled on ipv6 with admin,admin credential. while in the web config you could disable ssh, webpage disabled ssh only over ipv4.

-company fixed it in less than a month after my report (read very quickly)

-company gave me new firmware over http, without any kind of digital signature

-company never published the firmware on the website so i’m the only one who have this….
and it is not the first time since i had 1.02 (the most up to date according to the website) but after my report they gave me 1.06…

what should i do?
-nothing?
-public shaming?
-publish the firmware myself?

esteban • June 11, 2018 9:55 AM

@tyson

OpenWRT/LEDE do seem better about updates, in that they don’t stop releasing them so quickly. They’re not great: one still needs to download and flash an entire updated image, then reboot. I’d much prefer something like ‘apt’ where it can automatically upgrade and restart individual services.

I’m concerned about the process of upgrading a potentially infected router. They only hardware-based way to do it is (almost always) JTAG. The web/ssh based upgraders and TFTP fallbacks are implemented in software that could have been compromised to backdoor each new image.

Dave • June 11, 2018 10:17 AM

Unfortunately, even the security-conscious can’t protect themselves. I have the RT-N56U and have been waiting for a patch but the last one was on 2017/03/31: https://www.asus.com/us/Networking/RTN56U/HelpDesk_BIOS/

The product is otherwise working great, the industry seems to have a policy of ending support to force you to buy new products.

I guess I’ll have to install 3rd-party firmware and hope that addresses the vulnerabilities.

Clive Robinson • June 11, 2018 10:47 AM

@ Bruce,

Pretty much no one patches their routers, so the vulnerabilities have remained, even if they were fixed in new models from the same manufacturers.

This is a problem that is difficult to fix. A lot of people do not have access to their outward facing router because their service providers do not want them to have access. This includes major ISPs that have outsourced the script reading support staff that frequently go through meaningless rituals with users just to run up support costs billable.

This however is of intetest,

On May 30, the FBI seized control of toknowall.com, a critical VPNFilter command-and-control server. This is called “sinkholing,” and serves to disrupt a critical part of this system.

I’m supprised this works… Some experts will tell you how the “bad guys” can get around this with relative easy (I’ve even described how in the past). It is known that certain State Level APT attackers already do this, including those of better provinance as being Russian…

AnonDutch • June 11, 2018 12:22 PM

Is it just me that feels like getting rid of my computer, cable box, smartphone and everything else that’s connected, after reading yet another story like this?

Bauke Jan Douma • June 11, 2018 1:29 PM

“So if it won’t clear out the malware, why is the FBI asking us to reboot our routers? It’s mostly just to get a sense of how bad the problem is. The FBI now controls toknowall.com. When an infected router gets rebooted, it connects to that server to get fully reinfected, and when it does, the FBI will know. Rebooting will give it a better idea of how many devices out there are infected.”

Is this Bruce, or a well-meaning spokesperson for the glorious FBI?
Just wondering.

Richard Schwartz • June 11, 2018 1:37 PM

Re: “if your current one is on the list, it’s at least 10 years old”. This is not true. Maybe 5, but not 10. The initial list was mostly older models, but the current list of infected models includes a fair number of models that should not in any way be considered obsolete and ready to be thrown away, and are definitely beyond a trivial expense for many people.

justinacolmena • June 11, 2018 1:41 PM

When infected routers contact toknowall.com, they will no longer be contacting a server owned by the malware’s creators; instead, they’ll be contacting a server owned by the FBI.

Oh. Much better. That’s what we all wanted to begin with, isn’t it? I understand Russia and North Korea are allies now that Trump is in office, but yes, now it is becoming quite obvious that the FBI and FSB are on much better terms and working much more closely with each other than they have heretofore let on for the public.

Be that as it may, with Snowden’s sudden (apparently prearranged) stranding and subsequent asylum in Russia, we do need to focus on preserving fundamental civil rights within the U.S. under the U.S. Constitution, which does enumerate a fundamental right, rather than a privilege, to bear arms. Or even elsewhere, what about the rights of the armed “little green men” in Ukraine who lack the official recognition of their Russsian fatherland?

Tatütata • June 11, 2018 2:30 PM

If I count correctly, there are eleven different brands of routers affected.

Did the malware use the same vulnerability in all of them? Then they’re either using all the same buggy code, and/or routers have degenerated into a kind of white goods business, where private labels are slapped on fairly generic designs with only a couple of bells and whistles for differentiation.

Tatütata • June 11, 2018 2:35 PM

But if the G-men control the C&C server, doesn’t that technically allow them to inject their own cr*p on the affected devices?

Foreign devices are fair game for all descriptions of US-TLAs, but is there a doctrine allowing them to do that to US residents and/or citizens?

Zaphod • June 11, 2018 2:41 PM

Perhaps the learned folk here (we know who you all are) could let us know what hardware/software they use as their ‘home router’?

Whilst not in the above cohort, here I go; I removed the ISP router and use a Buffalo 600 with the most up to date DD-WRT image.

Zaphod

Ratio • June 11, 2018 3:00 PM

@Tatütata,

Did the malware use the same vulnerability in all of [the affected brands]?

Sounds like it:

There’s little doubt that whoever developed VPNFilter is an advanced group. Stage 1 infects devices running Busybox- and Linux-based firmware and is compiled for several CPU architectures. The primary purpose is to locate an attacker-controlled server on the Internet to receive a more fully featured second stage. Stage 1 locates the server by downloading an image from Photobucket.com and extracting an IP address from six integer values used for GPS latitude and longitude stored in the EXIF field. In the event the Photobucket download fails, stage 1 will try to download the image from toknowall[.]com.

Clive Robinson • June 11, 2018 4:16 PM

@ justinacolmena,

I understand Russia and North Korea are allies now that Trump is in office

Either you have not said what you mean, or you have not studied history.

The North Korean’s have been allies of both Russia and China since before the end of WWII. In fact China would almost certainly not be a “Communist” state if it had not been for what is now North Korea sending thousands of train loads of food weapons and battle hardened troops to support them.

The Korean war actually started as a push by Stalin against the US that were compleatly and uterly unprepared for what was to follow. The US might have had a chance if US politicos had alowed the commander in the field the nukes he demanded, but they did not. Even though the US killed atleast a third of what became North Korean citizens, the US was guarenteed to lose. As the Chinese leader at the time pointed out the standing Chinese army in 1950 actually outnumbered the number of US citizens. The US did not have the manpower or determination to win.

It’s a humiliation the US War Hawks have never been able to get over and in part it was responsable for the automated idiocy a decade or so later that gave rise to another humiliation the US it’s self and it’s citizens still can not come to terms with and that was the Vietnam war.

One of the few things both Russia and China agree on is that the North Korean problem the US Government has is very much of their own making.

It’s something that even US historians point out with little prompting. If left alone the North and South of Korea would have had some level of reunification two or more decades ago. Now what you might or might not agree on is what the result might or might not have been. But one thing is clear the North has man power and raw resources, the south the technology and importantly the scientific ability. If even only partial reunification happens the new Korea will be an economic power house that frightens not just the US, Japan and Taiwan but even Germany and Russia. China likewise is trying along with Russia to get some kind of benificial position with North Korea and even South Korea for their own economic protection.

It will be interesting to see what happens. Especially if as some economists believe that President Trumps tarrifs are just a gambit to get rid of tarrfis entirely.

If you are implying as many Western Newspapers are saying that Trump is cosying up to NK, personally I would say it’s been the first time in seventy years that the US has behaved as a “Rational Actor” towards Noeth Korea. The real question needs to be is why are the NK’s responding?

After all under the “Fool me once…” doctrine the NKs ought to be extreanly wary of anything the US Gov says, because past history shows the US will just reverse or rescind any agreement within six months…

Think on it if you like like Iran, the US set a series of conditions that they thought the Iranians would never agree to. Guess what Iran agreed and acted upon it, this was not the US game plan so instead the US State Dept and Gongress has thrown a hissy fit, and shown themselves up for what they realy are on the International Stage with the spot lights full on… Opps they’ve had their bluff called, and they have norhing to raise with…

As the ancient Chinese Curse has it for the US “May you live in interesting times…”

I would expect a whole lot more of this to follow… But as has been noted by others if either of the two original nominees had made it to the White House in all probability US citizens who are in uniform would be heading on mass back to the middle east on one pretext or another to become more cannon fodder… It appears that even the Israeli Defence Force recognise that would not be a good idea…

So yes “Interesting times”.

Clive Robinson • June 11, 2018 4:33 PM

@ Bruce,

It wouldn’t be surprising if the Russians targeted routers to build a network of infected computers for follow-on cyber operations. I’m sure many governments are doing the same.

You know it is OK to say that the US, UK, Auz, Canadian and NZ Signals Inteligence entities are “known to be doing this”. Likewise you can add the Dutch, Finnish, French, German, Swedish SigInt entities “are also known to be doing this” and a whole bunch of other countries including India, Singapore and the wider Atlantic-Eyes and Pacific Eyes are at it as well and that most of it’s results ends up at the NSA to be turned from raw to refined Intel.

Much as some IC people would like the information to not be in the public domain, it is and you can not put the genie back in the bottle.

Which should be an object lesson to those in the State Governments about “Privacy” and “Golden Frontdoors” and other NOBUS type stupidity. As used to be said and they should have been mindfull of,

    You reap what you sow

Clive Robinson • June 11, 2018 4:49 PM

@ Zaphod,

Nice to hear from you. With regards,

Perhaps the learned folk here (we know who you all are) could let us know what hardware/software they use as their ‘home router’?

As I’ve mentioned before my personal development etc machines I use are not protected by a router hardware or software. Because they are not connected in anyway to external networks.

As for other machines I’ve mentioned the data diodes I’ve designed, developed and constructed. Also that they are in effect “energy gapped” being inside safes inside a locked RF cage used at other times for high power RF design in the 5KW and up range.

I’ve even described how people might go about designing their own SCIF and using security end points that are “energy gapped” from any machines that are used as communications end points.

Complicated as it sounds the OpSec rules for such systems are way way less than for other designs. And at the end of the day as the old saying of “Lose lips sink ships” indicates, it’s frequently supposedly easier to use systems that have nearly unworkable OpSec Rules where info haemorrhages into the waiting ears of adversaries.

65535 • June 11, 2018 10:08 PM

This post is somewhat confusing and causes even more questions as certain posters have noted below. Can some of this questions be answered in a better way?

Cost problem of throwing away routers that are possibly not infected:

‘Re: “if your current one is on the list, it’s at least 10 years old”. This is not true. Maybe 5, but not 10. The initial list was mostly older models, but the current list of infected models includes a fair number of models that should not in any way be considered obsolete and ready to be thrown away, and are definitely beyond a trivial expense for many people.”’ -Richard Schwartz

I agree with Richard schwarts that taking a person’s asset or small business owner’s asset and tossing it in the rubbish heap is not very cost effective.
The list of 70+ routers and network attached devices have asset value it seems like a waste of money to the average Jane/Joe to junk:

Asus RT-AC66U (new)
Asus RT-N10 (new)
Asus RT-N10E (new)
Asus RT-N10U (new)
Asus RT-N56U (new)
Asus RT-N66U (new)

D-Link DES-1210-08P (new)
D-Link DIR-300 (new)
D-Link DIR-300A (new)
D-Link DSR-250N (new)
D-Link DSR-500N (new)
D-Link DSR-1000 (new)
D-Link DSR-1000N (new)

Huawei HG8245 (new)

Linksys E1200
Linksys E2500
Linksys E3000 (new)
Linksys E3200 (new)
Linksys E4200 (new)
Linksys RV082 (new)
Linksys WRVS4400N

MikroTik CCR1009 (new)
MikroTik CCR1016
MikroTik CCR1036
MikroTik CCR1072
MikroTik CRS109 (new)
MikroTik CRS112 (new)
MikroTik CRS125 (new)
MikroTik RB411 (new)
MikroTik RB450 (new)
MikroTik RB750 (new)
MikroTik RB911 (new)
MikroTik RB921 (new)
MikroTik RB941 (new)
MikroTik RB951 (new)
MikroTik RB952 (new)
MikroTik RB960 (new)
MikroTik RB962 (new)
MikroTik RB1100 (new)
MikroTik RB1200 (new)
MikroTik RB2011 (new)
MikroTik RB3011 (new)
MikroTik RB Groove (new)
MikroTik RB Omnitik (new)
MikroTik STX5 (new)

Netgear DG834 (new)
Netgear DGN1000 (new)
Netgear DGN2200
Netgear DGN3500 (new)
Netgear FVS318N (new)
Netgear MBRN3000 (new)
Netgear R6400
Netgear R7000
Netgear R8000
Netgear WNR1000
Netgear WNR2000
Netgear WNR2200 (new)
Netgear WNR4000 (new)
Netgear WNDR3700 (new)
Netgear WNDR4000 (new)
Netgear WNDR4300 (new)
Netgear WNDR4300-TN (new)
Netgear UTM50 (new)

QNAP TS251
QNAP TS439 Pro
Other QNAP NAS devices running QTS software

TP-Link R600VPN
TP-Link TL-WR741ND (new)
TP-Link TL-WR841N (new)

Ubiquiti NSM2 (new)
Ubiquiti PBE M5 (new)

Upvel Devices -unknown models (new)

ZTE Devices ZXHN H108N (new)

This is just the list of devices not the total number of devices from Symantec that is liked in the article. The number of devices could be quite high and costly to junk.

Soft-reset or hard reset question.

The FBI recommends a soft reboot. Symantec then explains to remove the malware a hard reset is necessary:

“Q: If Stage 1 of VPNFilter persists even after a reboot, is there any way of removing it?

“A: Yes. Performing a hard reset of the device, which restores factory settings, should wipe it clean and remove Stage 1. “-Symantec

“So if it won’t clear out the malware, why is the FBI asking us to reboot our routers? It’s mostly just to get a sense of how bad the problem is. The FBI now controls toknowall.com. When an infected router gets rebooted, it connects to that server to get fully reinfected, and when it does, the FBI will know. Rebooting will give it a better idea of how many devices out there are infected.” …”Is this Bruce, or a well-meaning spokesperson for the glorious FBI?”- Bauke Jan Douma

Why not just do a hard reset and get rid of stage one of VPNFilter since the device will only become reinfected and the FBI will just at your device to a list?

[From Wikipedia]

“Mitigation, Both Cisco and Symantec suggest that people who own affected devices do a factory reset. That is typically accomplished by using a small, pointed object, such as a straightened out paperclip, to push the small reset button on the back on the unit for 10 to 30 seconds (time varies by model). This will remove the malware, but also restores the router to all original settings.”-Wikipedia

https://en.wikipedia.org/wiki/VPNFilter#Mitigation

or

https://en.wikipedia.org/wiki/VPNFilter

Why not do it right the first time around rather than become a test subject of the FBI? Or, worse get reinfected with “ssler” as Cisco and Symantec note:

“A newly discovered (disclosed on June 6) Stage 3 module known as “ssler” is capable of intercepting all traffic going through the device via port 80, meaning the attackers can snoop on web traffic and also tamper with it to perform man-in-the-middle (MitM) attacks. Among its features is the capability to change HTTPS requests to ordinary HTTP requests, meaning data that is meant to be encrypted is sent insecurely. This can be used to harvest credentials and other sensitive information from the victim’s network. The discovery of this module is significant since it provides the attackers with a means of moving beyond the router and on to the victim’s network…. fourth Stage 3 module known as “dstr” (disclosed on June 6) adds a kill command to any Stage 2 module which lacks this feature. If executed, dstr will remove all traces of VPNFilter before bricking the device.”- Symantec blog

https://www.symantec.com/blogs/threat-intelligence/vpnfilter-iot-malware

Could the FBI control your router?

“…if the G-men [slang for FBI men] control the C&C server, doesn’t that technically allow them to inject their own cr*p on the affected devices? Foreign devices are fair game for all descriptions of US-TLAs, but is there a doctrine allowing them to do that to US residents and/or citizens?”- Tatütata

Given the FBI’s tainted status with a lot of people why trust the FBI at all?
Does the so called domain “ToKnowAll[Dot]com” sinkhole really work to stop the malware?

‘This however is of intetest, “On May 30, the FBI seized control of toknowall.com, a critical VPNFilter command-and-control server. This is called “sinkholing,” and serves to disrupt a critical part of this system.”… I’m supprised this works… Some experts will tell you how the “bad guys” can get around this with relative easy (I’ve even described how in the past). It is known that certain State Level APT attackers already do this, including those of better provinance as being Russian…”-Clive Robinson

Clive Robinson makes a reasonable point. Will not the malware C&C server be moved to a different or multiple domains?

Clive Robinson notes the problem on front facing router interfaces which only ISPs control:

“… A lot of people do not have access to their outward facing router because their service providers do not want them to have access. This includes major ISPs that have outsourced the script reading support staff that frequently go through meaningless rituals with users just to run up support costs billable.”- Clive Robinson

How does the average Jane/Joe go about securing the ISP facing and controlled interfaces without losing all internet connectivity?

Actual infection method maybe Busybox:

‘Sounds like it: “…little doubt that whoever developed VPNFilter is an advanced group. Stage 1 infects devices running Busybox- and Linux-based firmware and is compiled for several CPU architectures. The primary purpose is to locate an attacker-controlled server on the Internet to receive a more fully featured second stage. Stage 1 locates the server by downloading an image from Photobucket.com and extracting an IP address from six integer values used for GPS latitude and longitude stored in the EXIF field. In the event the Photobucket download fails, stage 1 will try to download the image from toknowall[.]com.”’-Ratio

See halfway down arstechnia:

https://arstechnica.com/information-technology/2018/05/hackers-infect-500000-consumer-routers-all-over-the-world-with-malware/

If Busybox is the infection method why not ID it and get to work fixing the Busybox problem?

me • June 12, 2018 2:20 AM

@K.S.
How do i publish cve?
information security is my hobby not job. i have found some vulns but i reported them to vendor (when it’s possible to contact them).
does anyone has a link that tell me what i should do?
Thanks

Clive Robinson • June 12, 2018 4:14 AM

@ esteban,

The web/ssh based upgraders and TFTP fallbacks are implemented in software that could have been compromised to backdoor each new image.

The usuall solution to this was signed code boot loaders, much loved by the “Walled Garden” fraternity, for their enrichment and supposadly your safety.

However as many have found the walled garden model is less effective than the “many eyes squash all bugs” FOSS ideal for stopping malware.

If you look back far enough on this blog you will find a conversation that started between @Nick P and myself about the fact that code signing was not secure. A number of others joined in some saying that it was the way forward. I listed reasons why it was not, and the usuall “Not Invented Here” / “Golden Goose” reasoning came back.

This was some time prior to stuxnet, that should have proved the point, but no apparently that was an exception for some reason… Since then all app stores have been hit by rougue code and every failing I listed has now been exploited by even the lowest form and least technical of attackers.

It’s just one of the many reasons I do not think “Auto-update” of code or even “manual download” are the sensible way to go, they are all way way to vulnerable.

However that raises the question of what to do about it, to which the answer is we don’t have a way…

That said the best method so far we have stoped using…

It was the mass produced CD/DVD on the front of magazines. Provided you purchased them from random places you were fairly safe as lots of people used to get their images that way and problems used to show up fairly quickly.

Now anyone upstream of you can intercept your packets and give you a fake download.

It’s a problem that realy requires us to solve the secure key distribution issue first and like Hard AI that’s been just round the corner for as long as our host @Bruce has been around.

Who? • June 12, 2018 5:14 AM

@ Clive Robinson

It was the mass produced CD/DVD on the front of magazines. Provided you purchased them from random places you were fairly safe as lots of people used to get their images that way and problems used to show up fairly quickly.

If we cannot trust on signed updates downloaded from external sources to our devices, either manually or automatically, how can we trust on the sources used to build these mass produced CDs and DVDs? People producing CDs for dead-tree magazines have no better chances to dump on them the right sources and binaries.

There had been cases in the past of mass produced CDs containing malware.

Who? • June 12, 2018 5:17 AM

Buying magazines on random places may help avoiding targeted surveillance, but as we know, the intelligence community is more interested in global surveillance these days. Less work, better results.

me • June 12, 2018 6:56 AM

@Who?
His point was not “you can’t backdoor mass produced cd”
his point was: “if you do so, someone will notice and there is no plausible deniability”

the intelligence community is more interested in global surveillance these days
True

Less work, better results.
I don’t think so

Clive Robinson • June 12, 2018 3:00 PM

@ Who?, me,

There had been cases in the past of mass produced CDs containing malware.

Yes they have, which is why the Mags that still have front cover DVDs tend to be somewhat paranoid in their testing. Way more so than most of us, and more importantly unlike most individuals they have considerable more and better resources than most of us. Including in some cases attracting help of the actuall developers to check every thing is as it should be.

As I said, nobody has come even close to a good way of doing software updates when you analyze it. That said however, of what there was and still is, on balance I think the Front Cover CD/DVD was thr best of the “working” bunch.

If you can think of ways to improve on it, now would be a good time to outline it. Because it’s becoming clearer to more and more people that what we have “is just not cutting the mustard”, and any suggestions good or bad may give others something to think about to implement or improve. As the old saying has it “Rome was not built in a day”… mostly it was brick by brick over centuries. Whilst I hope finding a reliable way to do software updates takes a few days, I know it’s going to take longer but hopefully not centuries.

bttb • June 12, 2018 5:21 PM

Somewhat OT, but malicious code signing and the Apple Macintosh:

“For almost 11 years, hackers have had an easy way to get macOS malware past the scrutiny of a host of third-party security tools by tricking them into believing the malicious wares were signed by Apple, researchers said Tuesday.
[…]
The technique worked using a binary format, alternatively known as a Fat or Universal file, that contained several files that were written for different CPUs used in Macs over the years, such as i386, x86_64, or PPC. Only the first so-called Mach-O file in the bundle had to be signed by Apple. At least eight third-party tools would show other non-signed executable code included in the same bundle as being signed by Apple, too. Affected third-party tools included VirusTotal, Google Santa, Facebook OSQuery, the Little Snitch Firewall, Yelp, OSXCollector, Carbon Black’s db Response, and several tools from Objective-See. Many companies and individuals rely on some of the tools to help implement whitelisting processes that permit only approved applications to be installed on a computer, while forbidding all others.
[…]
‘To be clear, this is not a vulnerability or bug in Apple’s code… basically just unclear/confusing documentation that led to people using their API incorrectly,’ Wardle told Ars. ‘Apple updated [its] documents to be more clear, and third-party developers just have to invoke the API with a more comprehensive flag (that was always available).'”
https://arstechnica.com/information-technology/2018/06/simple-technique-bypassed-macos-signature-checks-by-third-party-tools/

& more in-depth info:
https://www.okta.com/security-blog/2018/06/issues-around-third-party-apple-code-signing-checks/

Ratio • June 12, 2018 6:00 PM

@(unsigned) ~0 << 010 << 010 ^ ~0,

Will not the malware C&C server be moved to a different or multiple domains?

From FBI Seizes Control of Russian Botnet (three Squid posts ago):

If a victim reboots an infected router, the malicious plugins all disappear, and only the core malware code survives. That code is programmed to connect over the Internet to a command-and-control infrastructure set up by the hackers. First it checks for particular images hosted on Photobucket.com that held hidden information in the metadata. If it can’t find those images—which have indeed been removed from Photobucket—it turns to an emergency backup control point at the hard-coded web address ToKnowAll[.]com.

So how do you get the current (Stage 1 / core) malware talking to the new C&C infrastructure?

(I wouldn’t hold my breath waiting for an actual answer to this question from the Very Serious “I could do that drunk and blindfolded, in fact me and my mouth already have” People. Endless babble or soothing sounds of crickets are far more likely outcomes.)

If Busybox is the infection method why not ID it and get to work fixing the Busybox problem?

There are firmware updates available for some of the affected router models.

Clive Robinson • June 13, 2018 2:07 AM

@ 65535,

With regards the reboot, as long as there is core code on the infected device that executes then there would be the potential for it to establish a new C&C server. It depends on how cleverly they coded it.

What the FBI appears to be assuming is that the reboot code is not sufficiently well coded thus falls back to a default hard coded host name.

I’ve not seen an analysis of the code, and for obvious reasons [1] no one should go looking for a copy of any malware code, no matter how good their credentials.

So you would have to look at how the code might function from descriptions or hyperthetically.

Back when I described the “headless C&C” method years ago I indicated you would have to pick a service that was “To big to block” such as Google and it’s search engine, thus should always be visable to the malware.

Importantly the service has to have a database of messages that can be searched for by the service.

The trick involves your selection of what is a “one way function” to find a “control message” in a way that makes examination of the malware tell you nothing, and watching the malware execute as one instance will not tell you about what another instance will search for.

The other reason I picked Google was it alowed the botnet controlers to distance or decouple themselves from Google. That is some services go out and crawl the web. Thus the controler could post a comment on an open blog that Google would then put in it’s cache. There are sufficient open blogs out there that the controler would never have to use the same open blog twice. Which if the also used other well established practices would make them very difficult for even the Five Eyes to find but in effect impossible for just about everybody else.

Thus the part most likely that the malware writers have got wrong is being impatient or the equivalent of the one way function is unpredictable.

Oddly the first well known Worm written by the son of the senior NSA scientist suffered from “impatience” in that the replication rate was set to high, thus the chain reaction built up faster than the decay and the network got swamped out. Getting your K factor right without a feed back mechanism is actually very difficult. In the case of malware if you do add a feedback mechanism then the malware can be detected or defeted by it unless you put a lot of care into it. We saw another type of malware feedback mechanism fail badly when Marcus Hutchins found and registered a domain name.

The use of a fall back hardwired command server was such a mistake. It’s also a sign that the malware writers were “impatient”. They should not have done it. Now there is a clear indicator or big red flag available as to just which devices have malware. It was a big mistake to make…

It also suggests that there may have been other “preasures” on the malware writers in that there is say an important time line involved like the aniversary of a historical event to be “celebrated” for ideological reasons.

They might also have lacked confidence in their one way function for some reason. Getting them to work well also involves selecting the right K value without feedback. But also making the one way function effective is quite difficult. In effect you are designing a cryptographic system not to disimilar to a one way function with a trapdoor. The information to do it is in the public domain but you have to know about it and more importantly know how to “bolt the bits together so they fit”. It’s an area where there are PhDs to be earned for the right curious minds, but untill someone breaks ground it’s not likely to be investigated[2].

Or the FBI are just crossing their fingers and hoping there is a sufficient time delay between the reset and the controller establishing a new command message.

[1] See Marcy Wheeler’s comments about the FBI / DoJ on Marcus Hutchins,

https://www.emptywheel.net/2018/06/06/to-pre-empt-an-ass-handing-the-government-lards-on-problematic-new-charges-against-malwaretech/

Marcy has as @echo noted gone a little over the top with the naughty words, but I can understand why. Because it appears to be a rerun of the tactics used against Aaron Swartz that led to his suicide. Basically they have declared that if they decide you are guilty then they will keep throwing charges at you untill you run out of money and can nolonger defend yourself. It’s a legal game thats just one of the tricks of “Rights Stripping” and it’s one you can not win on the legal playing field. It’s the sort of thing you would expect of a tyrannical dictatorship trying to look like only a Police State. As I’ve said before you do not want to be a foreigner in the US when the FBI/DoJ decide to “show trial” you… As our host @Bruce noted many years ago, it’s counter productive. Because researchers will nolonger come to the US for conferences, thus academic conferences will move out or stay out of the US which will mean that in turn academic researchs and companies will move out of the US… Not smart but then the US legal system appears to be moving the “dumber and dumber” way these days. As my father pointed out to me many years ago “The place to be when there is trouble, is somewhere else”. The US is now “Trouble” for those involved with anything “cyber” and if you are “Jonny Foreigner” God help you.

[2] It’s the “Dark Arts taboo” issue. If you come up with the idea and publish a proof of concept, you are automatically evil. However once there is a “baddie” of sufficient evil intent we need “goodies” “To vanquish the evil doers and their works of evil”… So it’s OK to get into it once a “baddie” has appeared but not before.

Petr Špaček • June 13, 2018 3:26 AM

Disclaimer: I work for CZ.NIC, i.e. the manufacturer of device advertised below. The following text contains advertisement for routers with automatic updates. Feel free to flag this post as spam and delete it if it is deemed inappropriate for this forum.

Most vendors of super cheap routers have no motivation to maintain operating system for these boxes, there it simply not enough money behind it.

If you really want automatic updates and are prepared to pay for it have a look at “Turris Omnia” or “Turris MOX” routers. Turris routers come with OS based on OpenWRT and have automatic updates (which can be turned off, if you want).

Turris MOX – modular router with automatic updates:
https://www.indiegogo.com/projects/turris-mox-modular-open-source-router-security-computers#/

Turris Omnia – monolitic router with automatic updates:
https://omnia.turris.cz/en/

65535 • June 13, 2018 6:39 AM

@ Clive Robinson

“…regards the reboot, as long as there is core code on the infected device that executes then there would be the potential for it to establish a new C&C server. It depends on how cleverly they coded it. What the FBI appears to be assuming is that the reboot code is not sufficiently well coded thus falls back to a default hard coded host name.”- Clive R.

I agree with that.

It would appear the FBI feels the code can re-infect a device and then supposedly the world friendly FBI will monitor the spread of refection – or possibly test some of their malware to see how good it works infecting routers and stripping SSL.

There could be many permutation of this senairo with unknown outcomes to the average Jane/Joe. I would guess domians and IP address can be changed or spoofed.

I would play it safe and do a hard reset and hope for the best [I don’t have one of the 70+ makes of routers/devices on the list which is of some comfort].

“[1] no one should go looking for a copy of any malware code, no matter how good their credentials… See Marcy Wheeler’s comments about the FBI / DoJ on Marcus Hutchins,”- Clive R

I cannot disagree. I have read Emptywheel’s post on this MH arrest and I am not happy with how the FBI failed to turn on the recoder before his reading his rights and the charges. I am not liking the way the FBI keeps doubling down on their attempts to get a plea bargain. That is one of the reason’s I noted some people might view the FBI as tainted. I will say Brian Krebs feels the FBI has its man but who knows.

“…trick involves your selection of what is a “one way function” to find a “control message” in a way that makes examination of the malware tell you nothing, and watching the malware execute as one instance will not tell you about what another instance will search for. The other reason I picked Google was it alowed the botnet controlers to distance or decouple themselves from Google. That is some services go out and crawl the web…the controler could post a comment on an open blog that Google would then put in it’s cache.”-Clive R

That is a good trick. I think that there are a number of other ways of achieving the same goal. For example this could be just a test malware run by a powerful actor and the next strain of malware say, VPNFilter v2.01 could be much more effective and elusive.

“There are sufficient open blogs out there that the controler would never have to use the same open blog twice. Which if the also used other well established practices would make them very difficult for even the Five Eyes to find but in effect impossible for just about everybody else.”- Clive R

That is true. Further, there plenty of platforms that do similar things making the malware harder to stop. This method you describe could be mutated or multiplied in certain fashions. This also could up the malware game to a very unpleasant level.

“In effect you are designing a cryptographic system not to disimilar to a one way function with a trapdoor. The information to do it is in the public domain but you have to know about it and more importantly know how to “bolt the bits together so they fit”. It’s an area where there are PhDs to be earned for the right curious minds, but untill someone breaks ground it’s not likely to be investigated[2]… If you come up with the idea and publish a proof of concept, you are automatically evil. However once there is a “baddie” of sufficient evil intent we need “goodies” “To vanquish the evil doers and their works of evil”… So it’s OK to get into it once a “baddie” has appeared but not before.” -Clive

I concur. I hear you on the proof of concept and then getting automatically black listed.

Further, you have demonstrated how to mutated a system and probably have caused the other side to try and best you.

This is the problem somewhat like a double edged sword with both edges getting sharper. Now, add in some State actors, microprocessor maker with nasty code and more swords are better made and in the field. This malware game seems to continue to mushroom and grow.

Returning to the possible method of infection being BusyBox and Linux, are there any experts here that can comment on exact problem and how to mitigate it? Is it Busybox only or a fault in the chips used with Busybox?

bttb • June 13, 2018 8:54 AM

From emptywheel regarding cyber sanctions:

“Even as Trump was working hard to get Russia admitted back into the G-7, Treasury was preparing new cyber sanctions against a number of “Russian” entities. This appears to be an effort to apply sanctions for activities exploiting routers and other network infrastructure (activities that the US and its partners engage in too) that US-CERT released a warning about in April.”
https://www.emptywheel.net/2018/06/12/the-new-cyber-sanctions/

bttb • June 13, 2018 9:25 AM

Clive, above, wrote: “Marcy has as @echo noted gone a little over the top with the naughty words…”

‘Abundant tweets about civil liberties & national security. “Has a longer memory than an elephant & keeps more records than Jim Comey.” Legendary potty mouth.’
https://mobile.twitter.com/emptywheel

In addition, there may be some historical context that some readers may be missing, for example, afaik: https://www.emptywheel.net/2010/04/29/do-bloggers-suck-or-does-tradmed-just-suck-more/#comment-303850
or
https://shadowproof.com/2009/07/13/marcy-wheeler-says-blowjob-on-msnbc/
https://mobile.twitter.com/HowardKurtz/status/2619416491

Of course, nowadays actors say f?ck you on televised awards’ shows.

Asus Repair Center • August 7, 2018 6:51 AM

I Appreciate the detailed. I went through this post and it’s quite amazing for the user. I use Asus router and its have great feature and The Asus repair center help me a lot to setting up my router with greatly.

Emmajasmine • July 1, 2019 5:21 AM

The Cisco 887VA router with VDSL2/ADSL2+ over POTS with 802.11n ETSI Compliant C887VA-W-E-K9-RF is a Integrated Services Routers with high performance for broadband access in small offices and small branch-office or teleworker sites with performance required for concurrent services, including firewall, intrusion prevention, content filtering, and encryption for VPNs, optional 802.11g/n for mobility, and Quality-of-Service (QoS) features for optimizing voice and video applications. The Cisco 887 C887VA-K9-RF is a newer Cisco 880 Series which are fanless type providing a quiet, comfortable working environment in small offices.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.