That Bloomberg Supply-Chain-Hack Story

Back in October, Bloomberg reported that China has managed to install backdoors into server equipment that ended up in networks belonging to -- among others -- Apple and Amazon. Pretty much everybody has denied it (including the US DHS and the UK NCSC). Bloomberg has stood by its story -- and is still standing by it.

I don't think it's real. Yes, it's plausible. But first of all, if someone actually surreptitiously put malicious chips onto motherboards en masse, we would have seen a photo of the alleged chip already. And second, there are easier, more effective, and less obvious ways of adding backdoors to networking equipment.

Tags: , , , , ,

Posted on November 30, 2018 at 6:28 AM • 33 Comments

Comments

Ignatio Ramus • November 30, 2018 7:30 AM

Makes you wonder who the original 17 sources are (i.e. what nation they _really_ work for)

Frank • November 30, 2018 7:31 AM

"there are easier, more effective, and less obvious ways of adding backdoors to networking equipment."

Dictatorships don't do things in the easiest, most effective, or least obvious ways. This type of attack vector fits the personality of the govt doing the attack.

echo • November 30, 2018 7:48 AM

Bloombergs original article was palusible but after issues were discovered and Bloomberg gaslighted everyone with an obstructive denial I lost all confidence in their entire publication. I still read Bloomberg articles from time to time but even where I may agree with the article I cannot move past the loss of confidence. Perhaps I am being intolerant but I won't have a relationship on the basis Bloomberg expects and have the attitude "once a cheater always a cheater". Bloomberg now have to prove with a meaningful apology and actions that they have reformed.

@Frank

Personality of the entity is an interesting topic.

Sancho_P • November 30, 2018 7:57 AM

”… we would have seen a photo of the alleged chip already.” (@Bruce)

Isn’t that a funny thinking? A photo from the outside of an electronic part will hardly reveal what is inside.
Also, why would you trust a photo from an unnamed source more than the alleged words of an unnamed source?
(sorry, can’t read the WaPo: “Democracy dies in darkness”)

Wael • November 30, 2018 8:36 AM

[...] but the story itself was short on hard evidence of a supply-chain compromise ...

No evidence presented makes the story a baseless claim. Statements like: "anonymous sources say" are worthless.

we would have seen a photo of the alleged chip already.

Or any other technical piece of information that can be traced and investigated.

Serve The Home has a somewhat technical article on the issue. I say that because I saw a picture of a PC board and a few diagrams. I have not read the article, nor do I plan on reading it -- have other things to do.

Matt • November 30, 2018 9:32 AM

More than likely other intelligence agencies would have spies in China and gotten hold of the details of the Chinese spy chips to take advantage of the intelligence sources too. No one would want this exposed, not the companies, not China, not the US, not Russia, or anyone else with the details, so they all would have an interest in discrediting it assuming it is true. Snowden disclosed a big source of NSA intel is from piggybacking on other intelligence agencies operations.

Scared • November 30, 2018 9:37 AM

Why would this have to be an additional chip?
Couldn't it be changes to an existing chip; the silicon or metallization or code in a CPLD? The first two could be seen on some advanced Xray, but not the last one.

echo • November 30, 2018 9:45 AM

@Wael

As funny as the Bloomberg article was I'm not completely convinced by the article by Serve the Home. On the theoretical possibilities given what I have read from multiple sources including material discussing other attack capabilities I don't know what to think.

There's an article about a report by a group of ex NSA staff on the DNC hack. One half of the team supports an external hack. The other half supports an inside job.

Time to get out the clay tablets...

Clive Robinson • November 30, 2018 3:19 PM

@ Wael, echo

They'll be of limited use.

That depends on if you've had a good Greek Dinner, if you have then they would aid in making it a realy smashing event ;-)

Phaete • November 30, 2018 4:33 PM

Sad though, it seems no real accountability for the wasted time and paranoia/chinafobia these writers have caused.
Let some accountant put some dollar figures on that.

Clive Robinson • November 30, 2018 4:41 PM

@ Bruce,

if someone actually surreptitiously put malicious chips onto motherboards en masse, we would have seen a photo of the alleged chip already.

Whilst I do not wish to chuck fuel on Bloomberg's fire...

The technical question still arises as to if "en masse" is actually necessary? And if not just how few are needed?..

As I said originally the conventional wisdom was/is you interdict at most one or two steps at most up the supply chain from the customer. In part because that is the way the paperwork from the Ed Snowden trove said the SigInt entities did it. In part because it avoids the "en masse" issue that conventional wisdom said would be needed from the manufacturers end of the chain.

Personaly I think conventional wisdom is wrong on the last point. The likes of Apple etc would "bulk buy" thus their order would be known about at the factory. I've yet to see any argument to refute this point, in fact I've actually seen supporting evidence in other supply chains.

But further I don't think you need to put a chip in every board in an order. Simplisticaly each motherboard has two sets of interfaces, those that are untrusted and those that are trusted. All that would be required for a "single chip" attack would be using a trusted interface that will be connected to all the other trusted interfaces on all the other untampered with motherboards when they have been installed at the end customers server rooms.

There are realy only a couple of ways to find the presence of such a chip non destructively. The first would be by "visual inspection" to save time in the same manner as an Astronomers "blink box" that is used to find commets etc. The negatives could be from various energy types spectrum ranges and rapidly and compared with those from a known good board. However there are plenty of hiding places it could be put out of sight. The second way is that the chip would have various signitures that would differ from the correct part if you are looking in the right way at the right time. Thus the probability of detection of a signiture could be delayed beyond a reasonable "test time".

There is also the question of "abnormal communications", whilst known to engineers for what seems like forever and certainly predating WWII, it was apparently unknown to most in ITSec prior to BadBIOS made the news, that Shannon Channels come in all shapes and forms. Thus any energy source that can be modulated in some form can carry information off of a motherboard... Likewise any modulated signal in the area a motherboard is in can be picked up if there is a suitable sensor. Thus the components in a switch mode regulator could be selected to act as a trasmitter or receiver transducer and it does not need to be a chip...

The thing about the Bloomberg story is it does not need to be true, just possible. We saw this with BadBIOS the premise behind it was never confirmed. However the fact it was possible and had been in the news encoraged others to build systems to exploit the channels identified. Which means I suspect that more than a few people will now investigate how to poison a supply chain...

Phaete • November 30, 2018 6:31 PM

The story is just a myth with kernels of truth strung together by misunderstanding experts, riding a wave of xenophobia.

Extraordinary claims require extraordinary evidence

Not a single piece of evidence, except "They said so" and when you ask "them", they say they never said that.
Proof on the level of The Real Housewives, not on university level.

echo • November 30, 2018 6:50 PM

@Clive

The thing about the Bloomberg story is it does not need to be true, just possible. We saw this with BadBIOS the premise behind it was never confirmed. However the fact it was possible and had been in the news encoraged others to build systems to exploit the channels identified. Which means I suspect that more than a few people will now investigate how to poison a supply chain...

This kind of thing has happened before. Knowing or having confidence that something is possible can inspire events. I suspect this is the kind of phenomena behind road rage and drone attacks. The counter psychology is instititionalising such as a baby elephant changed to a stake in the ground. An adult elephant who could pull away without blinking never does because the elephant believes the chain and stake is immovable.

echo • November 30, 2018 7:45 PM

@Phaete

Proof on the level of The Real Housewives, not on university level.

This article and almost all the discussion surorunding it is led by men. In fact I cannot remember single woman's voice on this issue in the media. Given the behaviour of men in university as evidenced by the now rightfully closed Unilad magazine and the responsibilities housewives carry I do wonder if you are using the correct comparison!

Al • November 30, 2018 10:08 PM

The important thing we seem to be missing here is that a story has a source, the journalist who brought this ever so plausible hack forward should not be allowed to stick their head in the sand and we also need not to be denying the possibility. My grandpa always said: Even if a story might seem bogus, the story was made somehow, what was the real source of the story and what was the opportunity or the goal of this story? The goal, obviously, was reached, we are talking about it, just like flying saucers. What if the remote possibility existed and the hack was not just applied to the aforementioned hardware. What if this hack is actually still being implemented but shushed away or vehemently denied - ask yourself who benefits from that and why have we not seen a motherboard without that chip and with that chip? And what data was it actually capturing, if at all... There are so many loose ends to this story that in 50 years from now we either know the reporter was right or that we were right to ignore it. Remember what my grandpa said about plausibility

Clive Robinson • December 1, 2018 12:17 AM

@ Phaete,

Extraordinary claims require extraordinary evidence

Yes and no.

To deprive someone of their liberty or life, or to go to war, or to make scientific discovery of life in another part of the universe then yes, that level of proof is required.

But what about for your own defence? That is you are walking down the street and you see a crowd ahead, do you need extrodinary evidence you are in danger before you will cross thr stteet or turn into another street to take an alternative route?

Likewise do we need extrodinary evidence that burglers exist befor we put latches and locks on our gates, doors and windows.

What level of suspicion is required before you investigate something for your own piece of mind?

The article you quote says,

    An extraordinary claim is one which is not supported by the available, or ordinary, evidence. Support for such a claim must therefore come from newly observed evidence, or a new recognition of existing evidence, which is extraordinary.

That is "extraordinary claims" are "end point claims" that is "Definitely exists" or "Definitely does not exist", they are not to be used when there has been no investigation or hypothesis testing.

Thus if you were to make Bloomberg's claim then yes you would require some level of evidence, perhaps not "extrodinary evidence" but you would certainly require some evidence, which currently there does not appear to be any that we know of.

However from a defensive stand point you are asking "Could such attacks be feasable?" then you need to "reevaluate current understanding" and what underlies it.

Which is my point above, the prior and in some cases still current "conventional wisdom" was/is it was not possible. It can now be shown with a few moments thought that in part the conventional wisdom was/is bassed on a "assumption" that is either "to broad" or "false"... Thus either way we should reexamine our chain of reasoning.

The "too broad/false" assumption is that "It is not possible to mount a directed attack at a target from the manufacturers end of the supply chain". That is, at the factory you could NOT say that motherboard X was going to customer Y. That is actually an "extrodinary claim" in it's own right and easily disprovable when you consider "custom orders". Likewise it's fairly easy to say the same for other special orders such as "bulk orders" beyond a certain size.

So the conventional wisdom that "You can not know at the factory end of the supply chain" is a wrong assumption because it has a spectrum of answers related to "The type of order"...

So having shown that conventional wisdom is wrong in that point we need to ask does it have other "too broad / false" assumptions. The answer is unfortunately yes. Conventional wisdom also implys that a supply chain attack would need to effect "all motherboards". Again the reasoning is incorrect, it's assuming incorrectly the motherboards will be "used in issolation" not just from the outside world which is a semireasonable argument, but by false inference "issolated from all the other motherboards" as well... Which we now know for data centers is an unreasonable assumption.

Showing that "conventional wisdom" has these two false assumptions, what are the implications of saying they are wrong, that is what becomes possible?

Well on the valid assumption that some "cloud infrastructure", "social networking" and "search engine" entities public and more private use custom or bulk ordered parts. We can now see it is infact more likely than not that the factory knows who the motherboards are for directly or can make a probable guess based on where they are being shipped to.

Further we can go on and reason that the manufacturer is also very likely to know or can make a reasonable assumption about the motherboards likely use in a data center. That is the motherboards will be used "collectively" and not in issolation.

Which brings us onto the next more interesting part...

On the assumption the manufacture wants "plausable deniability" or just to be covert just how many motherboards do they need to tanper with in the range from "one to all"? Well it depends on your assumptions about "Goods Inwards Test" (GIT) and "Stock In Hand" (SIH) for spares etc.

But it's easy to see that GIT is not going to be a "Full" test because that would be a "destructive test" on all boards... Further due to time constraints GIT may not be a "Burn In Test" either but just a limited functional test on a fractionaly small subset of all the boards.

Thus it would be easy to design such an "implant chip" not to do anything for the first week or month of being powered up. Which would in most cases see it safely into a production rack if that is where it is destined for which is most probable. Thus the number of boards that would need to be implanted if all go into production is just one, or two for reliability.

So the next question is about SIH spares, things fail there is something called "The bathtub curve" you can look up for expected failures against time. The important initial part is the "juvenile failure rate", because that is what a full stress "Burn in Test" is about, the implant chip has to wait to get beyond this point before doing anything untoward and some boards will fail. So you need sufficient implanted motherboards to get to this point which might only be 1% of the order size. The customer will have an "expected service time operation" which means that they will replace the data center in three years or less and maybe five at the most. Thus the number of spares they keep on hand will be predicated by this and the manufacturers stated product life cycle which might only be a year and a half. Again it might only be a very small number, just a fraction of the order size.

The upshot is to be covert and have deniability the manufacture is only going to implant maybe three or four boards upto a very small fraction of the order size.

But they can also make any malware on such an implant chip a "toe hold only" device. That is it only infects one or two other motherboards that then go on in turn to infect one or two other motherboards each with a quite slow replication rate. This way if the attack is discovered it looks just like an already started covert attack from outside of the data center, unless extensive monitoring is in place the probability of finding the actuall original implant is very small indeed.

If I was going to do a covert attack via supply chain poisoning at any point a "toe hold" implant would be my prefered way to go about it. And if I was realy "thinking hinky" I would actually make the toe hold implant go after a network switch[1] as it's target to infect, with the infection in the switch then attacking the motherboards (but then I think nasty that way routinely ;-)

The final question though is having got the malware onto all the motherboards would it do you any good? The answer to that is actually quite difficult to say. That "conventional wisdom" is that such an attack would only be to "exfiltrate data", but as I've shown conventional wisdom has issues due to false assumptions, and the area of "payload function" is pure conjecture when it comes to both state and large corporate motivations.

We only have to look as stuxnet which was actually aimed at North Korea by the US and the more recent attacks on Saudi Oil interests to see that "data exfiltration" was not at all what the payloads were all about.

Thus again because I'm nasty that way I'll assume that the intent of such a toe hold implant chip would not be "data exfiltration" because it's going to get the implant chip found in fairly short order... which would be a waste of the entire effort for little gain[2].

No I'd be looking to build a cyber-doomsday device like a reverse kill switch etc. Consider the effect of taking out all US cloud services, social networking, search engines and much of the telecommunications network etc all at the same time of say midnight the day before "black Friday" or some other date. The real problem would not be the infiltration of the kill switch payload but synchronising the time the switch is thrown. Obviously this is a quite low bandwidth function thus quite covert... Whilst such an attack would not do much physical harm it's self, the secondary effects would be quite significant and something the US in particular are going to be sensitive to.

[1] Going after a network switch is actually easier than it sounds. Because in a large data center the types of switch in use are actually very limited in manufacturers/models and zeroday vulnerabilities tend to effect entire ranges of models, which is why the likes of the NSA like their big brother "routers" as a "target of choice"...

[2] The majority of malwares failings and why it gets found relatively easily is because it's not covert, it's noisy thus it get seen like mouse droppings and thus the traps come out and it gets the chop. We know that only some "state level" attacks are like this. However it has not stopped the assumption forming that "China = data theft" affecting "conventional wisdom"...

echo • December 1, 2018 1:11 AM

@Clive

If I was going to do a covert attack via supply chain poisoning at any point a "toe hold" implant would be my prefered way to go about it. And if I was realy "thinking hinky" I would actually make the toe hold implant go after a network switch[1] as it's target to infect, with the infection in the switch then attacking the motherboards (but then I think nasty that way routinely ;-)

I don't have a link handy but I read a new article yesterday of an attack on ISP supplied (?) routers which was used as an entry point to infect networks and domestic customers with malware or ransomware or something, I can' remember which. The reason this succeeded so easily is a lot of the routers on buyers lists are fairly standard and not updated very often.

Wesley Parish • December 1, 2018 2:43 AM

I found the Light Blue touchpaper article
Making sense of the Supermicro motherboard attack
https://www.lightbluetouchpaper.org/2018/10/05/making-sense-of-the-supermicro-motherboard-attack/
quite informative on the technology involved.

But they regard it as a difficult job:

Limited interception or modification of SPI communication is something that a medium complexity digital chip (a basic custom chip, or an off-the-shelf programmable CPLD) could do – but not to a great extent.

They even prpose a scenario for compromising the BMC:
In order to start its own Linux, AST2400 boots using the U-Boot bootloader. I noticed one of the options is for the AST2400 to pick up its Linux OS over the network (via TFTP or NFS). If (and it’s a substantial if) this is enabled in the AST2400 bootloader, it would not take a huge amount of modification to the SPI contents to divert the boot path so that the BMC fetched its firmware over the network (and potentially the Internet, subject to outbound firewalls)

As the article also points out, the BMC firmware is 32MB in size, and even though 32MB is no longer large in today's networks, it is still large enough to come to a reasonably competent Net Admin's attention, particularly coming from an unknown source.

Then another method of attack is mentioned:

But there’s another trick a bad BMC can do — it can simply read and write main memory once the machine is booted. The BMC is well-placed to do this, sitting on the PCI Express interconnect since it implements a basic graphics card. This means it potentially has access to large parts of system memory, and so all the data that might be stored on the server. Since the BMC also has access to the network, it’s feasible to exfiltrate that data over the Internet.

Which seems to me to be rather clumsy. Given the choice between stealing a gate and stealing a key, I'd leave stealing the gate to the likes of Samson. You can hide a key in your pocket, whereas you can't hide a gate in you pockets very easily. (I assume there are Houdinis out there who would make it a matter of pride to steal the whole fence as well. I leave it to them.) But then I'm lazy.

And I still want to know where I can get a microchip with seven-nautical-mile lithography.

Phaete • December 1, 2018 6:53 AM

@Clive Robinson,

The thing about the Bloomberg story is it does not need to be true, just possible.

However from a defensive stand point you are asking "Could such attacks be feasable?" then you need to "reevaluate current understanding" and what underlies it.

We've had several discussions about hollywood scenarios being used in IT scaremongering, i don't do them.
Is it possible to get hit by a falling piano, yes, will i wear anti falling piano hats, no.

But what about for your own defence? That is you are walking down the street and you see a crowd ahead, do you need extrodinary evidence you are in danger before you will cross thr stteet or turn into another street to take an alternative route?

Well, i actually will walk up to them and check what's up, must be something up (good or bad) when people congregate.
But the fact that you only listed 2 flee options (cross street or turn into another street) shows how different we process information.

I do agree broadly with your suggested use scenarios.

To fully understand your (very)chained rhetoric, i want to ask you the following question.

If next week we get a similar story, Chinese newspaper says Chinese secret service found malware in Intel chips, but no hard evidence at all, just their word, would you believe that story and defend it's possibilities just like the bloomberg story?

@echo

You caught me (mis)using the common (mis)perception instead of reality for comic effect. My bad, some of those ladies are far more trustworth then bloomberg

echo • December 1, 2018 8:30 AM

@Phaete

You caught me (mis)using the common (mis)perception instead of reality for comic effect. My bad, some of those ladies are far more trustworth then bloomberg

I was like "what?" then "ouch" then a penny slowly dropped. I finally got there.

Wesley Parish • December 3, 2018 3:33 AM

@echo, Wael, Clive Robinson

Clay tablets do not cure headaches, as they are quite hard to swallow. Much less stomachaches.

ParanoidByDesign • December 3, 2018 3:48 AM

What is this is "just" the Chinese getting their hands on the keys that exploit an existing backdoor ?
There are strong suspicions what amounts to backdoors exist in the consumer PC ME architecture, so it's not impossible it does for the BMC that is the ME's equivalent on server motherboards as well.
No need for adding hardware then, and it would explain everybody denying it, if those backdoors exist, nobody has any interest in admitting to them even if they didn't get an NSL about it.
I'm in conspiracy theory territory here but unfortunately with ITSEC reality sometimes surpasses fiction.

Wael • December 3, 2018 5:38 AM

@{Wesley Parish, echo, Clive Robinson},

Clay tablets do not cure headaches

Yes, they have limited use. Green light cures some headaches, though. It just has to be at the right intensity and frequency.

Clive Robinson • December 3, 2018 11:11 AM

@ Wesley Parish, echo Wael,

Clay tablets do not cure headaches, as they are quite hard to swallow. Much less stomachaches.

Oh you've probably never tried, Kaolin and morphine mixture[1]... My mother swore by it and Milk of Magnesia mixture as both were "Kill or Cure" ;-)

Kaolin is a form of clay and well morphine has all sorts of properties including making you constipated, which is one reason German U-Boot officers were encoraged to make use of it...

Sadly I'm told geting hold of K&M mixture in the US over the counter is well neigh impossible since Ronnie "the Ray Gun" Reagan was in the Whitehouse and Mrs Reagan unlike Queen Victoria "Was not amused" by it...

But if you grow lettucies at home in various growing dirts, the heart and root of a lettuce contains various amounts of natural laudinum which is in the same opiate family as morphine. So you could have "home grown" K&M if you wack it in a mortar and work it over with the pestal ;-)

[1] A UK medical note on K&M mixture, https://www.netdoctor.co.uk/medicines/digestion/a6945/kaolin-and-morphine-mixture/

Roman Zenka • December 3, 2018 2:25 PM

The Bloomberg article shows a beautiful photograph of the chip on a finger and next to a penny.
They do not seem to mention this was done for illustration purposes only. Are those photos fakes?

Wael • December 3, 2018 10:03 PM

@Clive Robinson,

the heart and root of a lettuce

I always eat the lettuce heart. Never felt any different; I just like the taste.

Wesley Parish • December 4, 2018 2:24 AM

@usual suspects

I've begun to dread the day when this Bloomberg article and the like are waved in front of the PRC's face and used as justification for 25% import tariffs on all Chinese rare earth goods. I also live in hope that at the same time 25% tariffs will be placed on all Chinese seven nautical-mile lithography Consumer Electronics until the PRC deigns to share such advanced technology with the United States of America ....

There's nothing quite like that last grand gesture, is there ....? (At least the seven nautical mile lithography will survive EMP strikes. I look forward to a certain someone asking the USN, the USMC and the USAF why they haven't upgraded to such technology yet. :)

Clive Robinson • December 4, 2018 4:05 AM

@ Wesley Parish,

One of the problems of a very dry mouth, is your tongue can get stuck very firmly in cheek 0:)

Mind you on the radio news yesterday evening was an item that apparently "Papa" had spoken to the naughty Xi and now all was sweetness and light in the kingdom...

https://www.bbc.co.uk/news/world-latin-america-46413196

Then I read the small print :-S

Drive By Idealogue • December 4, 2018 4:58 AM

I don't think it's real. Yes, it's plausible. But first of all, if someone actually surreptitiously put malicious chips onto motherboards en masse, we would have seen a photo of the alleged chip already.

Ahh, the Fake News call-out on S.O.S.

You know you can fit a small chip under a big chip right?

Wesley Parish • December 5, 2018 2:55 AM

@Clive Robinson

You are no doubt aware that microchips made with seven nautical mile lithography would be a substantial aid to making aircraft and ships and even motor vehicles stealthy? Upgrade a B52's electronics to multicore chips made with seven nautical mile lithography, and it becomes instantaneously stealthed. You would never guess that it was a long-range bomber after such an upgrade, would you? (It's true that there would be some side-effects, but such an upgraded B52 would also be immune to either directed or undirected EMP weapons.)

I guess though that to implement a proper seven nautical mile lithography process, the US silicon industry would have to upgrade from their cherished Olduvan to the more modern Chatelperronian techniques, and they may prefer to bang heads together rather than rock the boat.

Clive Robinson • December 5, 2018 7:28 AM

@ Wesley Parish,

It's true that there would be some side-effects, but such an upgraded B52 would also be immune to either directed or undirected EMP weapons.

Well at seven nautical miles per device you know it's never going to crash... Because one primary side-effect is it could never take off ;-)

The real question though is "How do you grow the crystals for the wafers without the moon bending the attempt?.."

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

← FBI Takes Down a Massive Advertising Fraud Ring Three-Rotor Enigma Machine Up for Auction Today →

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of IBM Resilient.