I’ve blogged twice about the Bloomberg story that China bugged Supermicro networking equipment destined to the US. We still don’t know if the story is true, although I am increasingly skeptical because of the lack of corroborating evidence to emerge.
We don’t know anything more, but this is the most comprehensive rebuttal of the story I have read.
Arclight • October 29, 2018 4:40 PM
The one thing I disagree with in that article is the notion that any medium or large enterprise would have baseband management and segmented networking configured in a way that would render a hardware implant ineffective.
Having worked with a large number of such customers, I don’t see a uniform level of competency or investment in non-revenue areas like security.
RealFakeNews • October 29, 2018 4:47 PM
Who would have motive/reason to attack Supermicro in this context? Who benefits from Supermicro being targeted by these rumors?
Jonathan • October 29, 2018 4:52 PM
Reading through Patrick Kennedy’s STH article, I agree with his sentiment. Clearly the author(s) of the Bloomberg article lacked understanding of their subject. It is unclear if any of the claims should be given serious consideration. That said, I wonder if Patrick has chosen to interpret some of the article’s ambiguities to fit his thesis.
Here are two examples.
1) A compromised BMC cannot “call home” because it sits behind a firewall or on an isolated network. Also, networking capabilities would make the Chinese chip much larger than claimed.
2) The Chinese chip does not have the I/O to attack CPU-to-DRAM communication.
I can see how a direct interpretation of the Bloomberg article could lead to these conundrums. But what if the author’s lack of technical knowledge is misleading Patrick to confusion.
Consider the following interpretation of the claims: The Chinese chip is placed on the board and connected to the SPI bus used by the BMC to manage the BIOS/OS image in flash. The chip uses this bus to manipulate the BIOS/OS image in flash. As part of the normal platform boot, the compromised BIOS/OS is loaded from flash to the CPU cache/DRAM and is executed. The compromised server CPU then opens a communication channel to support the call home functionality.
Notice that this attack manipulates the CPU execution without the need to attack DRAM. It also calls home without using the private network connected to the BMC, or through networking SW/HW integrated into the Chinese chip.
Jonathan • October 29, 2018 5:19 PM
A similar comment to mine (but more eloquently made) can be found here:
The Chinese chip doesn’t have to attack the server BIOS/OS directly. It can attack the boot of the BMC itself and then use the compromised and more powerful BMC to extend its attack on the BIOS/OS.
Yet another Bruce • October 29, 2018 5:23 PM
@Jonathan
Does this mean that the BMC boot loader typically does not require the main firmware image to be signed?
Jonathan • October 29, 2018 5:55 PM
@Bruce
That appears to be the case.
https://blog.eclypsium.com/2018/09/06/insecure-firmware-updates-in-server-management-systems/
Godfree Roberts • October 29, 2018 6:46 PM
The Bloomberg article is like thousands I’ve read over the past 40 years: it takes some factoid out of context, finds the most extreme projection, the worst interpretation, and makes the most outrageously negative predictions.
OK, that’s how Western media work in general, but here’s the thing: all of those articles about China’s economy have been wrong. There must be at least 35,000 of them (few with any originality) since 1975.
The Economist magazine alone published 56 predictions of a Chinese ‘hard landing’ in that time, for Christ’s sake. Every single one was completely wrong. 100%.
And to this day, no reputable Western writer has felt sufficiently curious to investigate this amazing phenomenon. Not one has said to her colleagues, “Hold on a minute. There’s a story here that could lift the lid off Western economics! The Chinese have always been right and we’ve always been wrong. It’s time to find out why”.
I suspect the reason she hasn’t written that story is That it has nothing to do with economics: it would lift the lid off Western propaganda and the lie of a ‘free press’.
Phaete • October 29, 2018 6:51 PM
Good stuff.
Factual, precise and unlike Bloomberg, a absolute minimum of ambiguity.
Clive Robinson • October 29, 2018 7:56 PM
@ Arclight,
I don’t see a uniform level of competency or investment in non-revenue areas like security.
I would agree, Oh and we know that Supermicro themselves are not exactly security concious… Otherwise why do they not use as a minimum “code signing” as has been pointed out a few times.
echo • October 29, 2018 10:14 PM
I am unhappy with UK media not being very educational and seeming to rely on pot stirring. I am very sceptical of US media who tend to blow the trumpet of the US but reading one economically orientated article the other day by Bloomberg their poorly written Supermicro articles had stuck in my mind. The net result for me is a massive loss of trust in often nationalistic media too eager to chase revenue.
I have witnessed with my own eyes many of the systemic failures other commentators have mentioned and provided examples and links to this.
“Gross professional negligence” is not always one big obvious failure but more a low level drift adding up to one big failure. This is something the courts recognise.
Honesty and accountability seem to be lacking in the UK. Not just security but “justice” are seen as a cost. This is made worse by too many UK lawyers chasing 60% margins and cherry picking cases for profitability even where this may be a breach of Convention rights and professional standards. It follows than anyone going light on security whichever form it takes may have more critical philosophical issues to address.
All of the above is why the #meetoo movement is essentially demanding a seatatthe table. Unless “security” and other forms of security like “justice” and human rights have a seat at the table the chance of compromised decisions is too high placing long term viability at risk.
Arclight • October 30, 2018 12:00 AM
@Clive Robinson
Not only are discount x86 PC vendors unlikely to spend any more than necessary on basic security, but the definition of a cloud server is pretty much a system that WILL be allowed to do a lot of talking on the Internet, probably via built-in gigabit interfaces.
There is plenty of opportunity for misconfiguration of security controls, firmware settings and firewalls that would allow some type of phone-home functionality.
Gunter Königsmann • October 30, 2018 1:01 AM
@Jonathan: A device with network access and an physical ethernet cable would definitively have been detected.
But a chip that claims to be a pci peripheral that comes with its own uefi extensions or windows drivers only needs to have 4pins and a few Megabytes of flash. An usb device claiming not to be a memory key but to contain signed windows drivers embedded into hardware would need to provide 4 Pins. Many Mainboards offer pads you can solder pci or usb devices to if you know it. Either for future extensions or for debugging/testing/programming. Remember BadUSB that allowed to use an USB device in order to reprogram the USB chipset’s firmware in order to use DMA in order to change a program in the system’s RAM?
Also a JTAG flasher that can replace the firmware of nearly every single chip or change arbitrary registers/pin states can be limited to 6 pins and most devices offer JTAG pads for testing purposes, often in a limited space small enough to solder a chip on it.
If the uefi or operating system is compromised one can use the system’s ethernet connection. If a firewall filters out traffic initiated by manipulated boards I am not sure: Most of the time it filters ingoing connections instead. And perhaps the cloud-based firms didn’t lie: perhaps the manipulated boards used windows drivers as an attack vector but if there were an windows installation it would run in an VM that never saw the fake devices trying to compromise it.
What speaks against all this: If they manipulated a firmware: Why would they leave the manipulating device lying around? For undoing eventual firmware updates? But wouldn’t that be detected, as well?
Wesley Parish • October 30, 2018 5:03 AM
@Gunter Königsmann
A chip that claims to be a (embedded) USB device would talk to the rest of the motherboard through the USB controller on the motherboard. And that would then talk to the memory through the DMA chip and other memory management chips, and to hard drive through the PCI bus. Which isn’t what has been claimed.
You could not place such a (embedded) USB device between the CPU and the UEFI (or ROMBIOS) and expect it to do anything. Or between the CPU and anything else for that matter, and expect it to do anything except take up space. The story is talking about something eitehr between the BMC and the rest of the motherboard, or between the CPU and persistent memory (or main storage, which is not necessarily the same thing.).
In which case I fully concur with Patrick Kennedy that it is implausible, and reiterate my fervent wish to find the sort of manufacturer who can manufacture microchips with traces only seven nautical mile wide. 🙂 I think they went out of business when Stonehenge didn’t prove all that portable. 🙂
Clive Robinson • October 30, 2018 6:25 AM
@ All,
I think we more or less all agree that for the specifics, few that there are, the Bloomberg story is at best implausible and if it were in a Scottish Court it would be “Case unproven”.
They only two named technical sources are Joe Fitzpatrick in the first Bloomberg story and Yossi Applebaum in the second. Although both distanced them selves from what Bloomberg had done what they said in theor own seperate posts is not unreasonable.
In fact Yossi Applebaum made an important point which was in effect “don’t throw the baby out with the bath water”. That is although the Bloomberg stories are probably close to being as fictitious as you are going to get, “hardware implants” are not just technically possible, we kbow they can be done.
Anyone remember BadBIOS from October 2013? the story was broken to the world by the Register and almost imediately got cold water poured on it by all and sundry. However a few of us on this blog pointed out that some of what Dragos Ruiu said was entirely possible and I had in the past used all the parts that would make it possible. Both myselve and RobertT made it clear that it was laptop speakers and mikes you would need to use as the old desktop moving coil devices were not suitable. I even pointed out how the original method of getting device drivers from ROM chips on IO boards was still functioning in windows. Well as we know in the following December two researchers Michael Hanspach and Michael Goetz published a paper of their findings to the Journal of Communication showing that a low bit rate acoustic channel as described on this blog was not just possible but they had built one. So that atleast proved that part.
Then now the path was made “bl33ding obvious” “everyone and his dog” developed acoustic systems for advertising crapware.
Then less than a couple of years later Lenovo were caught using exactly the IO device driver mechanism I had pointed out to put “persistent crapware” into their consumer grade laptops. Which created quite a stink at the time.
The take away is like that pointed out by Arthur C Clarke many years ago,
And,
The latter is important because as we saw with information on the NSA TAO ANT catalogue, it realy does not have to be very advanced at all to be magic. It just needs to be outside your scope of thinking.
One of the problems with many researchers is they have a lot of depth but next to no real breadth. Likewise “tech talking heads” have a great deal breadth but it is increadably shallow. In both cases they entorely miss what many engineers especialy those that are a little “elderly but distinguished” and have both good breadth and depth know from experience works, and can build relatively quickly and easily with work they have already done.
As Yossi Applebaum has pointed out do not make the mistake of thinking this sort of attack is not possible simply because a couple of Bloomberg reporters in “bonus seeking mode” pushed out a story for which they could offer no evidence and mis reported what they were told.
We know that the NSA has had hardware implants from atleast a decade ago, we know they have used them, we know from GCHQ behaviour with regards The Guardian Laptop motherboard which got photographed that it’s not just the obvious Flash ROM chips where firmware can hide. It takes not much imagination to realise that for various reasons of “deniability” a well funded SigInt entity would use atleast two seperate attack vectors that you would need to align to pull off a successfully “plausably deniable” attack, thus they would put in four to six attack vectors for ensuring toe holds exist to get back in. Why? Well because when you sit down and actually plan how you as a super power nation state SigInt entity would ensure a continued intel supply you don’t realy have other options…
Clive Robinson • October 30, 2018 10:53 AM
@ ALL,
I kind of missed it but it’s actually BadBIOS’s aniversary it all blew up five years ago this week…
Oh and as for Proof of Concept, I guess you could say this was published,
https://www.schneier.com/blog/archives/2013/11/badbios.html#c2244046
A month before Michael Hanspach and Michael Goetz…
Yes folks you read it here first 😉
Any way in the UK it’s that time when we ask “Witch or Guy or both” get to feal the flames” 😉
echo • October 30, 2018 8:00 PM
@Clive
I completely forgot about Halloween. I wanted to do something. I even have a gothic vampire style outfit I pulled together. It looks nice but fishnets on cold nights like this? I would have to wear thigh high boots over the things to keep warm. Talk about being an idiot magnet! The reason I’m stuck with this is I didn’t have the money when Laura Ashley long velvet gothic style dresses were clogging Ebay. One of these would have been amazing with a Russian style fur hat and a decent tailored long coat.
I want to visit a decent barbecue not be stuck inside in a noisy room listening to electro music with drunk 21 year olds with ants in their pants. Unless I can “socially engineer” a “mark” its a night in with Cup-a-soup and a granny cardigan.
MarkH • October 31, 2018 3:21 AM
I’m indebted to the commenters on this thread. I haven’t been following this story (except to take note of broad skepticism).
I didn’t know what a BMC is … this stuff is worlds away from my work. What a God-awful security liability such setups are!
Whatever the truth concerning SuperMicro, BMCs are obviously a perfect attack target.
It’ll be interesting, if we can learn anything about malicious exploits in the wild…
dennis • October 31, 2018 7:30 AM
Wow seems there is a lot wrong with the bloomberg report.
But regarding the feasibility of attacks only through a 6-pin chip, it isn’t as limited as that rebuttal story makes it seem like.
You don’t need live interception of hundreds of wires to be able to mess with the system. There are many smaller serial transmissions used in modern machines that sometimes have surprising levels of access. And no, they don’t need to be as complex as USB. You also don’t need networking on the chip when you can control your NIC through a simple bus.
Just to give two examples:
https://labs.portcullis.co.uk/blog/graham-gsuberland-sutherlands-44con-presentation/
Anyway, I still wonder why an extra chip is required at all. Updating the firmware of some included chip sounds like an easier attack vector.
GregW • October 31, 2018 7:40 AM
Operation Mockingbird redux?
Logical • October 31, 2018 7:43 AM
China would do said spying if they could. They can. Therefore, they did. The current leadership of China has different policies, more aggressive and controlling than past leaders. This type of risk taking to gain a political or military advantage is entirely plausible.
WhiskersInMenlo • October 31, 2018 12:37 PM
I wonder who is pulling a leg?
https://boingboing.net/2018/10/27/someones-lying.html
At this point if a batch of systems destined for an interesting
customer had been altered it is likely that no one is willing to talk.
Some might be gagged by FISA warrants but that is not the only way to
silence issues of national security.
The potential for manipulation of the story to manipulate trade with China
is non zero.
https://slate.com/human-interest/2013/11/u-s-in-world-war-ii-how-the-navy-broke-japanese-codes-before-midway.
Is this story equivalent to “desalinization plant was out of order”?
Suffice to say that motherboards and more need to be designed to thwart
this risk.
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
AC • October 29, 2018 4:40 PM
The Risky Business podcast also tore a strip out of Bloomberg. Episode 517.