IRONCHEF: NSA Exploit of the Day

Today’s item from the NSA’s Tailored Access Operations (TAO) group implant catalog is IRONCHEF:

IRONCHEF

(TS//SI//REL) IRONCHEF provides access persistence to target systems by exploiting the motherboard BIOS and utilizing System Management Mode (SMM) to communicate with a hardware implant that provides two-way RF communication.

(TS//SI//REL) This technique supports the HP Proliant 380DL G5 server, onto which a hardware implant has been installed that communicates over the I2C Interface (WAGONBED).

(TS//SI//REL) Through interdiction, IRONCHEF, a software CNE implant and the hardware implant are installed onto the system. If the software CNE implant is removed from the target machine, IRONCHEF is used to access the machine, determine the reason for removal of the software, and then reinstall the software from a listening post to the target system.

Status: Ready for Immediate Delivery

Unit Cost: $0

Page, with graphics, is here. General information about TAO and the catalog is here.

“CNE” stands for Computer Network Exfiltration. “Through interdiction” presumably means that the NSA has to physically intercept the computer while in transit to insert the hardware/software implant.

In the comments, feel free to discuss how the exploit works, how we might detect it, how it has probably been improved since the catalog entry in 2008, and so on.

The plan is to post one of these a day for the next couple of months.

Tags: , , , , , , ,

Posted on January 3, 2014 at 12:20 PM • 67 Comments

Comments

Petréa Mitchell • January 3, 2014 12:42 PM

Given the vintage of the server named, I presume the namer was a fan of Iron Chef USA rather than the original Iron Chef.

I don’t know if this provides any insight into how the exploit was developed, but Iron Chef in all its incarnations is a cooking competition where two chefs have an hour to make a multi-course meal using a theme ingredient which is revealed at the start of the hour. Usually it’s a guest challenger vs. one from a group of regulars who specialize in different cuisines.

Wally • January 3, 2014 12:44 PM

It has been reported that NSA used techniques to intercept hardware in transit to implant their malware on their servers.

One wonders if similar dirty tricks have been used in the sales of the hardware. Suppose CIA/NSA learns that a rival intelligence agency is looking for new servers. It would be easy for them to subsidise the price of servers, such as the HP Proliant 380 DL G6 so that this would be the most economical choice for them. All it would take is the co-operation or understandin of the sales department of HP. Perhaps the lower price of some server brands can be explained by subsidies by the NSA.

Luigi Rosa • January 3, 2014 12:47 PM

The slide linked says that IRONCHEF supports DL380 G5, not G6 as written in the article (probably a typo).

Maybe they have upgraded IRONCHEF, but DL380 G5 are now servers for museums.

Steven • January 3, 2014 12:49 PM

You know, I’d kind of like to know more about “interdiction”. I mean, presumably they aren’t hijacking USP trucks on the interstate. So what are they doing? And who else is in on it?

edge • January 3, 2014 1:02 PM

I’m very concerned that these physical attacks are on server class machines. When I first read that they were intercepting computers in shipping I assumed they were doing it to personal computers of targeted suspects.

But these server machines are meant to be used by a whole lot of people. I wonder what (if any) minimization procedures are used in these kinds of attacks. Furthermore, these machines are probably owned by innocent third party organizations who wouldn’t appreciate this sort of trespass.

I don’t like this.

John • January 3, 2014 1:12 PM

I am totally pissed. Working at a facility that has an air gapped network to protect the critical infrastructure, now I have to start checking all these systems for the HOWLERMONKEY and IRONCHEF components, as well as looking for encrypted UDP packets containing RC6 constants is not what I ever expected to have to protect my employers network from my own government.

Anyone make snort filters for this yet?

Bruce Schneier • January 3, 2014 1:16 PM

“Anyone make snort filters for this yet?”

I think this is an excellent idea.

Daniel • January 3, 2014 1:20 PM

Since this attack features a hardware implant the best way to detect it is with a physical inspection. So the NSA’s working assumption is that most companies that purchase servers never open them up and inspect their insides or if they do look inside, such as to change a HD, they don’t pay any attention to the rest of the hardware inside.

To my mind the best way to foil these hardware implants is to simply make hardware inspection a regular part of one’s anti-surveillance routine.

@Chirs. The problem with an RF scanner is that the it can be easily foiled. I’d assume that the RF hardware implant only transmits within a narrow time window and spends most of the time asleep.

Daniel • January 3, 2014 1:31 PM

Re: snort.

I’m not convinced that is going to be useful. There are two different software layers in play. One software that runs the CNE and another software than runs the RF transmitter. The RF is two-way. So it could be that the CNE software is sending unencrypted traffic. What makes me think that? The fact that the data sheet anticipates the target uninstalling the CNE software. This suggest to me that the CNE software is actually something that mimics some type of ad-ware or other program that the end-user is likely to perceive as innocuous (not NSA spying).

G. Bailey • January 3, 2014 1:32 PM

It sounds like a SNORT filter will be nearly useless for this. IRONCHEF is only used to persist the software component for the technique, while the hardware implant does the communication via “MRRF” or “GSM”.

SNORT can only see the traffic on the legitimate airgapped network, not the stuff transmitted by the hardware implant.

G. Bailey • January 3, 2014 1:35 PM

@Daniel

I doubt that hardware inspection will work. I suspect that the way that the hardware is installed is that one of the many ICs on the board is replaced with a malicious version, i.e. via soldering. It’s possible that an X-ray analysis with a “known good” board would work, but there would likely be false positives.

Daniel • January 3, 2014 1:46 PM

@G. Bailey.

You could be correct, there is so little we know. But its worth realizing that the smaller the RF transmitter (a) the more heat it will generate and (b) the shorter range it will have.

Since it is anticipated that the two-way capability of the RF chip will be used that means the smaller the chip the shorter the distance which means the closer to the server the agent must be. There is a real on-the-ground difference between having an agent sit in the car 300 meters away on the side of a busy road reinstalling the CNE software and requiring the agent to have physical access to the premise. At some point in time why even bother installing the hardware in the first place if the NSA has that type of free physical access to the server farm?

So while you could be correct my instinct tells me that the RF chip is likely to be big enough that it pushes the signal a fair distance and thus it could be caught by a physical inspection. I could be wrong about that. Without having more technical details we are both just guessing.

Another Kevin • January 3, 2014 2:28 PM

One of these a day for THE NEXT COUPLE OF MONTHS?

Gadzooks.

At least now we know what President Bush was up to in 2007 with his signing statement that construed the Postal Reform Act as authorizing the opening of Americans’ first-class mail and parcels in exigent circumstances for foreign intelligence collection. http://www.nbcnews.com/id/16472777/ And it’s considerably broader in scope than even I, a cynic about governmental usurpation of power, imagined.

Ben Brockert • January 3, 2014 2:41 PM

Do you remember watching Enemy of the State, and to make Gene Hackman’s character seem paranoid they had him work inside an elaborate Faraday cage?

Now I’m not sure how you could possibly do that would show someone as being paranoid.

Nick P • January 3, 2014 2:41 PM

There are two solutions to these kinds of risks:

  1. EMSEC (TEMPEST)
  2. TSCM

Emanation security focuses on ensuring that passive or active EM eavesdropping is a non issue. As this is an implant, the EMSEC defense would be a shielded room or operating the PC’s in a shielded safe. Think of it as a server rack that traps signals in. Protection would also be necessary on the power line and any data cables leaving the site.

TSCM is basically bugsweeping & countersurveillance gear. Spectrum analysis tools come to mind. The tools would scan wireless spectrum looking for anything that shouldn’t be there. If this is combined with a shielded room & no wireless devices policy, the bugs would literally be the only thing transmitting with any real power. The problems with the TSCM aspect are that (a) there’s a variety of spectrum for bugs to use, (b) equipment/expertise is limited, and (c) as another commenter pointed out the bug won’t be transmitting continuously.

So, if we must use possibly interdicted devices, I’m thinking that DIY EMSEC rooms or server containers with TEMPEST style zoning are becoming a better idea all the time. Room should be soundproof, too, to defeat emanation attacks using sound.

Henrik Holst • January 3, 2014 2:45 PM

@Daniel

Well they could have access to the server facility as in them simply renting a rack at Rackspace, InterXion and so on. If so they would be close enough to alot of servers to use their RF while not having to have complete physical access (which might raise suspicions).

Roxanne • January 3, 2014 2:48 PM

Ben (above) had the same thought as I did. I wonder how many people (and businesses) are retro-fitting their offices to be Faraday cages? Will this be a new growth industry for home and office remodeling?

Brian M. • January 3, 2014 2:53 PM

@G. Bailey:
I doubt that hardware inspection will work. I suspect that the way that the hardware is installed is that one of the many ICs on the board is replaced with a malicious version, i.e. via soldering. It’s possible that an X-ray analysis with a “known good” board would work, but there would likely be false positives.

Once upon a time, I worked as a hardware lab tech for an OEM. When a chip is replaced on the board, you can’t tell it’s been replaced. Period. That is normal quality work, and I’m sure that the NSA has some very skilled technicians.

SMM has been a big target since it was introduced, so I’m not surprised that the NSA has been using it.

Since the NSA has been intercepting hardware shipments, then checking varios BIOS data for corruption after you receive the system won’t be effective. Plus they will also intercept the target’s attempts to download new firmware, and redirect the target to their own servers.

A way around this would be for many people to post checksums and versions of their machine’s firmware. Then it could be possible to spot differences in what are supposedly the same firmware versions, or see if someone is running a version that was never issued by the manufacturer.

As for network traffic, it really is questionable about how the traffic is exfiltrated. Is the data obviously encrypted, or is it hidden within something? I suppose it depends on the target. There’s a lot of stuff that can be stashed away in, say, a DNS request. The user’s activity would have to be correlated to the Internet traffic. For instance, how many times does a web page want http://massive-garbage-here.cloudfront.com, and you just bob your head and agree to the access? If you didn’t correlate your browser traffic, from your browser, to what is going out your router, something like that would slip by in a heartbeat.

The traffic is coming from inside the target’s machine. Thus, you would have to have complete logs, like through Fiddler or equivalent, and cross-reference that to a Snort or Wireshark session running on another machine. When you see traffic that doesn’t match normal system or user traffic, then you have a suspect.

kevinm • January 3, 2014 3:17 PM

EMSEC? The RF link need only reach the nearest still infected IRONCHEF equipped server to download a new payload and re-infect the host. To be 100% protected you would need a Faraday cage per server.

Daniel • January 3, 2014 3:29 PM

Re: EMSEC

and there is another issue with that “solution”. The initial plant from the interdiction. The RF is only there as a back up once the CNE has been uninstalled by the target. But even if there was a Faraday cage for every server there would still be a time window between the interdiction and the discovery of the CNE when the system was compromised and transmitting data back the the NSA. So all the EMSEC does is prevent the /reinstallation/ of the CNE and doesn’t do anything to prevent data leakage from the initial attack.

npcomplete • January 3, 2014 4:10 PM

@Brian M.

Yes, I would also recommend checking the BIOS image checksum of your system. One problem is that most BIOS use the very weak CRC16 or CRC32 algorithm and that can easily be manipulated to give the right result.

Typically SMM code executes without the OS’s knowledge, but I think you can still detect its execution. It still requires IRQ 9, the same interrupt used for ACPI calls, to switch, and one could possibly trap those SMM mode switches by replacing the routine pointed to in the CPU’s (local apic) vector table.

npcomplete • January 3, 2014 4:24 PM

actually to update my post, yes, comparing checksums using your own hash algorithm of the manufacturer’s image, and then extracting that image from the BIOS would work, only if the firmware tools (by AMI, etc) actually give you the same bits when you save/backup your system bios. I was initially thinking of using comparing the checksum supplied in delivered image and the system’s.

DB • January 3, 2014 4:37 PM

Since this is based on two things: (1) a BIOS component, and (2) a hardware transmitter component, working together…. one way to defeat it is to wipe out the BIOS and upgrade it to your own compiled fully free and open source variant, like http://www.coreboot.org/. This could possibly disable the attack even if the hardware transmitter were hard to spot and physically remained.

No closed source code can be considered “safe” in today’s world. Open source can be audited by various security professionals (note: that does not guarantee that it HAS BEEN, only that it CAN BE).

Checksum/fingerprinting of a manufacturer’s BIOS image is not enough, when they can be served with secret court orders forcing them to undermine the constitutional rights of every customer, as they have been doing to every telecom. The solution is more openness, not more secrecy.

As a warning: as long as Bruce keeps posting similar blog posts to this one, I intend on posting similar responses to this one.

Sam Greenfield • January 3, 2014 4:56 PM

Luigi Rosa, the date on the slide is either 2008-07-14 (top right) or 2007-01-08 (lower right); the DL380 G5 was the most recent model of the DL380 at the time. c.f. http://www.pcmag.com/article2/0,2817,2271014,00.asp and http://www.bladewatch.com/2009/09/04/how-old-is-my-server/ The G6 wasn’t released until 2009.

It’s weird to me that the slide gets the name of the machine wrong (380DL versus DL380). I can’t think of anyone working regularly with these machines making this kind of mistake.

JRB • January 3, 2014 4:58 PM

I’m an EE. There’s a few misconceptions floating around here about the nature of RF emissions and electronics.

Unfortunately monitoring the spectrum for unexpected emissions would be extremely difficult. Accurate RF measurements (for FCC certification) are done in purpose built laboratories with expensive equipment, often outdoors and in remote areas. Looking for a rogue signal emitted from a device as complex as a computer is fraught with difficulties: the RF emissions from a computer are wideband, full of signals and spikes, and it is constantly changing. Properly working devices are allowed to briefly emit more RF energy than the nominal limit, so even a sudden peak is not necessarily a red flag. Even with an RF analysis would not find a rogue emission unless you had some idea of what to look for, and probably a known good machine for comparison. Good luck.

As others have said, hardware inspection will not work. In additional to the difficulties of even spotting a replaced part, the I2C bus they use requires only two signal lines, and then power and ground, so one could even disguise the rogue RF device as a four wire passive component. A case fan is another attractive possibility.

As for testing BIOS firmware checksums – if the hardware is compromised, there would ways around that.

If you are extremely concerned about evading these attacks, I’d consider purchasing components separately and assembling them. The Faraday cage may be an option as well, although they are less perfect than is commonly thought. A poorly built Faraday cage is just a tinfoil hat. A DIY EMSEC solution is probably not going to cut it.

It seems that we will need to begin to treat hardware itself as an insecure environment.

Nick P • January 3, 2014 5:08 PM

@ Kevinm, Daniel

The EMSEC solution is to the plethora of emanation/RF attacks the NSA is doing. It should prevent some. The BIOS side of the problem is fixed by BIOS security approaches I posted in last thread. The interdiction problem is a supply chain issue. The software exploit problem has yet other solutions. And so on.

There’s no one-technique-fixes-all approach to system security (esp against NSA). There will be a set of methods working together to minimize issues. It goes without saying that any solution I post to a specific issue (eg RF) is intended to be complementary with other solutions to other issues.

re enclosure per server

Amend my previous post to say we need to develop those too. That’s essentially what a TEMPEST-shielded server is anyway. I’d like a 1U, 2U and 4U chassis that does the same for generic servers. Minimal config is a shielded power port, management port, and [untrusted] data port.

Steven C. • January 3, 2014 5:46 PM

“$0” is so cheap that you’d think it cost-effective to bug every server coming out of the factory. In fact maybe that’s what it implies – if all servers have the implant they may only need to identify it such as by Ethernet MAC to communicate with it – no further cost. You could scan its barcodes in person, in customs, sniff an Ethernet address from an already-compromised network, or from seeing an IPv6 SLAAC address used on the public Internet.

These servers’ serial numbers[1] vary in the third letter according to the assembly site, which may be in the Americas, Europe or Asia-Pacific region. Even if not all servers are bugged, I think some will be more likely than others…

  1. http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?sp4ts.oid=1121516&spf_p.tpst=kbDocDisplay&spf_p.prp_kbDocDisplay=wsrp-navigationalState%3DdocId%253Demr_na-c00866744-23%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

Iain Moffat • January 3, 2014 7:23 PM

Bruce asked for comments on how it works and might be detected. The wording of the description suggests that code running on the main CPU in SMM accesses the RF module using I2C. This might lead one to look for an extra I2C device (using i2cdetect under linux for example). I suspect it might be more complicated (and harder to find) than that on a DL380G5.

HP Proliant motherboards have an I2C sensor bus for temperature and voltage monitoring controlled by the embedded iLO (HP Proprietary equivalent of IPMI) processor and accessed through it. This is necessary so the iLO can access the sensors when the main processor is not operational. From what I remember (or can still find on the HP website) of the DL380 G5 the iLO is only accessible from the host using HP provided tools so even if the BIOS implant runs on the DL380 CPU in SMM mode and the RF module is controlled using I2C, the implant may do so via the iLO microprocessor and there is nothing even a live CD operating system would find from running i2cdetect on the main CPU. There is quite a lot on the lm_sensor list relevant to this topic (e.g. http://lists.lm-sensors.org/pipermail/lm-sensors/2007-January/018641.html )

We really need someone who still has access to a 380 G5 to run i2cdetect ( http://www.lm-sensors.org/wiki/man/i2cdetect ) under linux and have a good look at the motherboard for a 3-pin i2c connector. I havent found it documented for the 380 G5 but some Proliants had an I2C 3-pin header for connecting smart array disc controller cards to the I2C bus – if present that would be an obvious place to connect the RF module. The service manual for the DL380 G5 (HP part 403167-404 is easy to google) has a diagram of motherboard connectors on p84 of the 13th edition, but it doesnt show that level of detail unfortunately.

Hope This Helps

Iain

Steven C. • January 3, 2014 8:33 PM

@Iain: I have access right now to a DL380 G5 running NetBSD 6, and I could have tried the i2cscan tool, but strangely there is no supported IIC (i2c) bus detected. I wonder if it’s available to the host at all?

The only sensors known to envstat are amdtemp0 (via PCI?), and many more temperature values from ipmi0 (being polled from the real i2c sensors by the service processor, I guess).

Steven C. • January 3, 2014 9:11 PM

How might the implant communicate effectively from inside of an electrically earthed steel box (possible among a stack of others)? Does MRRF simply mean ‘medium range RF’? Or if GSM is used, doesn’t that imply it must register with a celltower (i.e. transmit) in order to receive anything? Also depending on there being coverage at that location in the appropriate GSM band.

ISTR the backplane for the hotswap disk drives has an i2c connection to it (to control LEDs and such?). That seems like the most ‘open’ area for a weak radio signal to get in/out, and easily communicate with other machines along the aisle of a datacentre.

I wonder what is the weird object visible in photo [1] but not photo [2]:
1. http://www.the620guy.com/Images/ebay/2012-12-12/2%20-%20HP%20ML370%20G5%20BP%20412736-001%203.JPG
2. http://www.auctiva.com/hostedimages/showimage.aspx?gid=1388322&ppid=1122&image=512812622&images=512812622,512812745&formats=0,0&format=0

Clive Robinson • January 3, 2014 9:54 PM

@ JRB,

    Properly working devices are allowed to briefly emit more RF energy than the nominal limit, so even a sudden peak is not necessarily a red flag.

The reality is likely to be a lot worse than this.

To save the cost of pasive filtering components, some years ago somebody came up with the idea of “whitening” the “spurs” via Direct Sequence Spread Spectrum (DSSS) techniques to fit “under the mask” (on older systems you had a BIOS setting to turn it on/off).

Now I don’t know what coding gain the currently have, but 20dB reduction would be fairly easily achivable for major spurs.

The interesting thing is you can knowing the spread sequence and chip rate build a receiver to lock on and despread the spurs back to the “unspread state”, which would give a range increase for a receiver of the square root of the coding gain (ie CG dB/2 so with 20dB = 100 you would get 10 times the noise floor range…).

Further by using simple inversion of the spreading sequence with the data (ie Phase Reverse Keying) you can leak data out covertly on all the spurs.

Thus you don’t need a transmitter chip just access to the DSSS sequence generator (which the BIOS has).

Whilst you could with the right equipment and knowledge find such data modulation, you would have to expect it’s existance in the first place…

I’ve mentioned the security implications of the DSSS “whitening” in PC’s a number of times in the past on this and other blogs going back several years (both bad and good, for instance on the good side I’ve outlined how it can be used to hide drift in the CPU clock appearing in network time stamps when looking to prevent crackers enumerating PCs via correlating the time delta on different VM instances to detect “Honeynets”).

So I suspect that quite a few people aside from myself have been aware of it’s potential for over a decade.

Daniel • January 3, 2014 10:06 PM

@Clive and JRB

One thing I do not have a good handle on is just how good NSA can shrink RF transmitters, as my experience is with them on a commercial scale. But my experience with them on that scale is that the tiny transmitters, such as the one in my laptop, have a limited range. I also believe that there are FCC regulations in this regard too, though I cannot imagine that such regulations would stop the NSA.

So when I hear JRB talk about setting up wires in a case fan (a rather ingenious place I might add) my thinking is that such a low-powered system cannot have any transmission range at all. One meter? Five meters? Or am I just wrong about that and the NSA can shrink a 300 meter transmitter down to the size of a grain of sand?

Figureitout • January 3, 2014 10:39 PM

Daniel
–The FCC enforcement has gone down quite a bit for radio since the “Red Scare” days. More likely a pissed off ham will find you first and report you. Unless you’ve got persistent blaring signals, or jam some interesting targets, you’ll probably be fine. I try to follow all FCC rules BTW. 🙂

And check out QRP for an idea of what’s possible:

http://en.wikipedia.org/wiki/QRP_operation

Clive Robinson • January 3, 2014 10:59 PM

@ Daniel,

    One thing I do not have a good handle on is just how good NSA can shrink RF transmitters, as my experience is with them on a commercial scale.

How small do you want to make it?

An oscillator can be made with a single inverter gate, by connecting the output back to the input via some form of tuned circuit, which does not have to have a high Q (contrary to many peoples belifes). To modulate it the simplest thing to do is apply the data in reduced form to the input of the gate, this causes the bias thus the switching point to change and thus phase modulate the signal. Slightly more complicated is to feed the inverter output into one input of a two input XOR gate and the data into the other this phase reverse modulates the signal and has quite a few advantages over other forms of modulation.

Quite a few years ago I used a 74LS XOR 140in DIP, a thirty MHz overtone XTAL an electret mic a few turns of copper wire and a couple of ceramic capacitors and by using two of the XOR gates as a push-pull arangment to drive a tuned circuit at 90MHz made an FM bug. With a six inch piece of wire as the antenna it could be heard over 50meters away. However when coupled into a coax line up to a folded dipole ontop of a house it could be picked up easily half a KM away with a portable radio with an NBFM mode.

The point is it’s not realy about size or power but how you couple the power into free space. The more effficiently you do it the greater the range. Further the higher the frequency you use the easier it is to make a physicaly small but efficient antenna.

Most modern logic as you are aware can run happily into the low end of the microwave bands thus printed circuit coils and trace antennas can be made quite efficiently.

But if you want to do it properly get one of those USB mobile broadband modems and take the case off, the entire transmitter receiver and modulator all fit in a tiny chip and the antenna is a small paper clip sized piece of wire folded up in the tip. These work quite happily at quite high data rates over four or five KM.

Clive Robinson • January 3, 2014 11:00 PM

@ Daniel,

    One thing I do not have a good handle on is just how good NSA can shrink RF transmitters, as my experience is with them on a commercial scale.

How small do you want to make it?

An oscillator can be made with a single inverter gate, by connecting the output back to the input via some form of tuned circuit, which does not have to have a high Q (contrary to many peoples belifes). To modulate it the simplest thing to do is apply the data in reduced form to the input of the gate, this causes the bias thus the switching point to change and thus phase modulate the signal. Slightly more complicated is to feed the inverter output into one input of a two input XOR gate and the data into the other this phase reverse modulates the signal and has quite a few advantages over other forms of modulation.

Quite a few years ago I used a 74LS XOR 140in DIP, a thirty MHz overtone XTAL an electret mic a few turns of copper wire and a couple of ceramic capacitors and by using two of the XOR gates as a push-pull arangment to drive a tuned circuit at 90MHz made an FM bug. With a six inch piece of wire as the antenna it could be heard over 50meters away. However when coupled into a coax line up to a folded dipole ontop of a house it could be picked up easily half a KM away with a portable radio with an NBFM mode.

The point is it’s not realy about size or power but how you couple the power into free space. The more effficiently you do it the greater the range. Further the higher the frequency you use the easier it is to make a physicaly small but efficient antenna.

Most modern logic as you are aware can run happily into the low end of the microwave bands thus printed circuit coils and trace antennas can be made quite efficiently.

But if you want to do it properly get one of those USB mobile broadband modems and take the case off, the entire transmitter receiver and modulator all fit in a tiny chip and the antenna is a small paper clip sized piece of wire folded up in the tip. These work quite happily at quite high data rates over four or five KM.

65535 • January 4, 2014 1:47 AM

@ Steven

“I’d kind of like to know more about “interdiction”. I mean, presumably they aren’t hijacking USP trucks on the interstate…”

The Feds don’t have to. It’s in the UPS’s fine print when you sign that little white form.

@ I am totally pissed. Working at a facility that has an air gapped network to protect the critical infrastructure, now I have to start checking all these systems for the HOWLERMONKEY and IRONCHEF components… encrypted UDP packets… not what I ever expected to have to protect my employers network from my own government.”

I agree. This is getting ugly. I have to start asking my hosting provider if they use HP Proliant servers anywhere in the chain – including cloud storage.

@ Daniel

“I’m not convinced that [SNORT] is going to be useful. There are two different software layers in play. One software that runs the CNE and software than runs the RF transmitter.”

Yes. That is a real problem!

@ G. Bailey

“SNORT can only see the traffic on the legitimate airgapped network, not the stuff transmitted by the hardware implant.”

That’s the problem with redundant types of spyware.

@ Another Kevin

You have found another piece of “semi-legal” spying puzzle – the warrant-less opening of US mail.

[Your link to NBC]

‘“But in signing a postal bill just before Christmas, President Bush said federal law also gives the government authority to open the mail “for Foreign Intelligence Collection.”

‘The full signing statement said:’

‘“The executive branch shall construe subsection 404(c) of title 39, as enacted by subsection 1010(e) of the act, which provides for opening of an item of a class of mail otherwise sealed against inspection, in a manner consistent, to the maximum extent permissible, with the need to conduct searches in exigent circumstances, such as to protect human life and safety against hazardous materials, and the need for Physical Searches specifically authorized by law for Foreign Intelligence Collection.”’

http://www.nbcnews.com/id/16472777/

@ Nick P; Steven C; Clive; JRB and others,

It’s going to take a combination fixes. But, Clive has me worried. This could get very expensive.

@ Iain Moffat

“You can find used ex-MoD tempest computer cases quite cheaply…”

That is a good catch.

But, how do you ensure that it gets to you and has not been tampered?

How do you get around the “interdiction” problem or re-insertion via agency “bag jobs”? With a large facility a bag job by one of many possible moles would counter fixes that occurred.

If you have to continually check each and every device for implants it will become an expensive issue. It is looking like “interdiction” is a huge problem.

Any ideas to counter the “interdiction” issue?

Wael • January 4, 2014 2:26 AM

Clearly it’s an intractable problem to solve. For two main reasons, lack of control and lack of awareness. Awareness is becoming “better”, it would seem. Control part? Not so sure, because at the end of the day, you’ll have to trust a group of humans (manufacturer, supply chain, OS, peripherals — these aren’t all humans, per se, but there are humans behind them). Was discussed a year or so ago before all this became widely known, in the C-v-P and subversion discussions

Rob • January 4, 2014 3:48 AM

No matter what the component is, an antenna would be required to do the ‘air gap’ transmission of data. The antenna and support circuitry could be designed into the board, however, I doubt such is the case.

I’d look for blue-wire modifications on the server board.

anonymous • January 4, 2014 12:48 PM

You cannot “print” computer parts. Why reinvent the wheel? There are plenty of open source hardware designs around, use one of those. As I said in another thread, Raspberry Pi, Beaglebone, MarS Board, Marsboard, etc.

camurgo • January 4, 2014 2:33 PM

@anon, from what I’ve been reading there are already 3D printers capable printing basic electronic components at very small scale; entire complex computer parts might soon be possible too. I agree that reinventing the wheel isn’t a good idea, I realized that soon after posting that comment…

Lance Cottrell • January 4, 2014 7:36 PM

The possibility of hardware intercept and modification is why I always worry a bit when I see some computer gear I have ordered being drop shipped directly from the factory in China.

In my more paranoid moments, I imagine there being two bins in the shipping department, one for normal delivery, and one for “special” recipients.

5050 • January 4, 2014 9:30 PM

The iLO 2 manual is available online, the microprocessor has a security override switch (p. 42), the iLO 2 also supports all IPMI v.2 commands. For install, unplug, open server, switch the secuirty override switch for the iLO 2 chip, load RF device, power on – with no iLO 2 security to install desired “iLO 2” firmware (or program using remote console features) – power off, switch the override switch back, close server and power on. Seems like iLO 2 event logs might log some actions. Linking to the iLO 2 microprocessor logical (out-of-band control too)?

Iain Moffat • January 5, 2014 4:59 AM

@65535 – I bought the Tempest cases to use a computer alongside HF radio equipment in an electrically quiet location so infosec wasnt a concern at the time. These things generally emerge in bulk lots from the MoD via tender sales so you have the option to turn up and collect with a truck – although I can see that doesn’t address all interdiction issues. In the UK the main outlet is Witham Specialist Vehicles (www.mod-sales.com) who were also selling “communications shelters” up to 20 foot container size for a few hundred pounds earlier this decade, which are effectively faraday cages for less than the cost of building one.

Iain Moffat • January 5, 2014 5:20 AM

Regarding the RF module and antennas – the DL380 G5 by its shape, form and acoustic output is not likely to be found on or under a desk. It is designed to be rack-mounted and racks are usually in dedicated machine rooms, often in datacentres. So any radio communication with a controller outside the building needs to escape from the DL380 case (metal and designed to pass FCC EMC testing), the rack (metal) and the datacentre (quite likely metal framed and metal clad). To me this would point to a fairly high operating frequency (short wavelength) where the holes and slots in the rack and case are less of a barrier (as a general rule any hole or slot much smaller than 1/4 wave is an effective shield or reflector and anything much bigger than a few wavelengths is no longer a barrier). I would also expect fairly high power especially from the outside in to overcome local electrical noise at the target end, unless the intent was just to jump firewalls or air gaps between two compromised servers within the same installation. Most EMC testing for radiated emissions covers 30MHz to 1GHz (or that was the case when I was involved in EMC early 1990s) so I suspect that a higher operating frequency would be needed to operate in a range where the DL380 was never expected to be RF-proof and thus escape the gaps in the DL380 case with any degree of efficiency.

The one exception to this is if the transmitter is somehow coupled to an unshielded external cable, in which case the cable is then able to operate as an antenna outside the case. In this case the shielding effects of both the DL380 case and the rack are bypassed but there will likely be extra wires visible inside the DL380. The optimum frequencies for that mode will be ones where the cable is between 1/4 and 1 wavelength long – given that power and network cables are usually a few metres that would point to low VHF. If I were doing it I would probably look at coupling to one of the network RJ45 jacks for that purpose.

Hope this Helps

Iain

Andy Lutomirski • January 5, 2014 8:26 AM

It would help a lot if Intel just got rid of SMM entirely (or at least added an option to verifiably kill it). Yes, it’s vaguely useful for UEFI secure variables, but every other use that I’m aware of is just a silly hack to maybe save a few cents on a better EC.

Killing off SMM would improve real-time performance, too.

Buck • January 5, 2014 9:25 AM

May be time to take another look at wireless devices that have been so much taken for granted… It could provide a nice plausible deniability to target vulnerable devices through their legitimate communication channels and then commandeer wireless chips for use in a more covert manner. Think of alarm systems, cell phones, smart meters… Hell, with the latter, one could modulate the power lines such that an RF implant hiding inside your case would only need to RX from a very short distance to the power supply.

Steven C. • January 5, 2014 1:11 PM

Just realised that my server is actually the DL385 G2 (AMD) not the DL380 G5 (Intel, with SMM). So my above comments about not being able to detect the I2O bus are erroneous.

I still think the hot-swap bays would be a good opening in which to put a transmitter. The backplane behind the drive bays is clad to form a mostly grounded plane except for some air holes, enclosing the rest of the server behind it. Anything mounted on the front of that PCB ought to be able to communicate effectively with an aisle of other racked servers. I believe the Molex connector on that board is for the system’s I2O bus (used to toggle the disk identification LEDs I think).

The document suggests the bug only connects to a covert network, for communication with another compromised device on-site. I guess that’s what MRRF refers to; but I do wonder about mention of GSM.

Mr. Pragma • January 5, 2014 4:00 PM

As Bruce said the real hram done – and problem – is trust being destroyed.

An example: One might create a bios-checksums.org project where millions enter some basic data about their mainboard and some BIOS checksum.

Now, whenever someone had some mainboard whose checksum was different, one could be reasonably suspicious or, the other way round, one could be relieved.

Problem is that after the nsa leaks nobody trusts nothing anymore. Soon enough there would be (possibly even justified) rumours about nsa/gchq/whatever being behind bios-checksums.org.

Which, albeit a somewhat dumb and contrived example, leads to a (imo) very important point Bruce didn’t even indicate (afaik):

Trust destroyed -> feeling of being helpless.

Which, by and in itself, very much plays into nsa’s & Co. hands.

As a sidenote, I’m glad about those posts because I feel, there is a way too strong – and not helpful – focus on encryption.

Looking at it, it turns out (imo) that nsa’s – by far – most powerful resource is outside of nsa. It’s things like complexity (of hardware and software), sloppyness, dumb (evidently misplaced) trust in the likes of M$ (Windows) or Dell and generally a wide spread “They’ll take care of me being O.K.”, whoever “they” may be.

THIS is the most powerful resource of nsa and not quantum computer (PR bullsh*t bingo) or their otherwordly cryptology capabilities.

RobertT • January 5, 2014 5:39 PM

From what I’ve read I’d be looking for RF comms to be local (just within the computer room) bridging air-gapped to internet connected within the same room. This assumes that both air-gapped and non air-gapped computers are co-located (same room) which has been the case for most commercial systems that I have seen.

I’d love to comment on the exact system workings but …..Tweet tweet tweet… there’s that annoying little birdy again.

Clive Robinson • January 5, 2014 6:53 PM

@ Robert T,

    From what I’ve read I’d be looking for RF comms to be local (just within the computer room) bridging air-gapped to internet connected within the same room

It’s a logical –but not necesarily correct– assumption to use as a starting point.

However it is also logical to assume that their are going to be more than two machines “within the same room”, which has a significant implication.

Specificaly the issue of colissions with transmissions and maintaining a covert presence.

There are three basic ways to stop TX colisions,

1, Time division multiplexing.
2, Frequency division multiplexing.
3, Code / sequency division multiplexing.

Of the three the first is the least covert and the third the most covert.

Likewise similar assumptions can be made about likely modulation types due to data rates etc. However this needs to be tempered by “receiver capability” in the presence of noise as well as minimising silicon real-estate.

Whilst there are quite complex modulation systems available those involving amplitude variation are not particularly covert and do not do well in strong interferance conditions. Likewise those that contain simple frequency modulation are not particularly covert but tend to perform marginaly better than those using amplitude variation. Thus my money would be on a phase modulation system which is more covert of which phase reverse keying tends to perform better in strong interferance.

My prefrence would be a DS-SS system which phase reversed the chip sequence with the data being Manchester or equivalent encoded.

Daniel • January 5, 2014 8:53 PM

Mr. Pragma.

You are part of the way there. The biggest thing the NSA has going for them it not FUD but power. Not the intellectual power of techies but the power of the military and the power of the jails.

FUD doesn’t have much effect unless it’s backed up by the threat of legitimate force. Sure, there are always the few who are scared of their own shadow or the rattle of bones by the witch doctor. But the real fear is what happens with the use and abuse of power.

What the NSA can collect is important but it’s only important to the degree that the government uses it to harm someone. And when one thinks about who the government can harm it is obvious that not all players are created equal. The Chinese government hacker doesn’t give a shit about the what the NSA can do because she knows that even if the NSA can identify her with precision they are not going to send a crack team to eliminate her. On the other hand the NSA can pass the data on to the FBI and put the domestic trouble-maker in jail. They can pass the data on to third world country and take out the local agitator who is bothering the authorities.

So I’d argue that behind the lack of trust in technology is a more fundamental lack of trust in and a consequent fear of power.

RobertT • January 5, 2014 11:32 PM

@Clive Robinson
“My prefrence would be a DS-SS system which phase reversed the chip sequence with the data being Manchester or equivalent encoded

Not sure that I agree.

In reality I’m guessing that the NSA would want to leverage proven RF hardware modules as much as possible. The most ubiquitous short range RF module around today is probably BlueTooth, there are plenty of very low power BT chips available so it would be easy to use the BT protocol BUT simply change the frequency. If you did BT protocol in the 433Mhz ISM band then the silicon would be easy to get working and all the comms protocols and negotiation would be done on the chip. The only difference is that their BT would be running at a different frequency so no one else could find it accidentally. All that NSA would need to do is to develop an ISM band transceiver supporting the correct BT modulation, Actually I think Maxim already has some weird catalog parts that would be suitable. If you supported both 433Mhz and 2450Mhz (normal BT) than you could communicate between the machines at one frequency and communicate with a compromised cell-phone at the normal BT frequency.

65535 • January 6, 2014 12:56 AM

@ Iain Moffat

Interesting bit on the Witham Specialist Vehicles – but in the states you would need a good excuse to use and/or disguise those vehicles, tents, or F-cages. Various law enforcement are interested in civilian use of used military equipment. And, still not all interdiction scenarios are solved.

A-Team • January 6, 2014 7:02 AM

“The plan is to post one of these a day for the next couple of months.”

Nice idea but doesn’t go far enough. There are 118 code words in those documents, of which 107 were completely new — not previously known from earlier Snowden documents or LinkedIn job ads. They’re all listed, explained and linked now at http://electrospaces.blogspot.com/

ALTEREGO, ANGRYNEIGHBOR, ARKSTREAM, BANANAGLEE, BLINDDATE, BULLDOZER, CANDYGRAM, CHIMNEYPOOL, COMMONDEER, CONJECTURE, COTTONMOUTH-I, COTTONMOUTH-II, COTTONMOUTH-III, CROSSBEAM, CRUMPET , CTX4000, CYCLONE Hx9, DANDERSPRITZ, DEITYBOUNCE, DOGCOLLAR, DROPMIRE, DROPOUTJEEP, EBSR, ENTOURAGE, FEEDTROUGH , FIREWALK, FLUXBABBITT, FORESTPLACE, FREEFLOW, GALAXY, GCNI, GECKO II, GENESIS, GENIE-compliant, GINSU, GODSURGE, GOPHERSET, GOTHAM, GOURMETTROUGH, GSMCM-II, HALLUXWATER, HAMMERMILL, HEADWATER, HOLLOWPOINT, HOWLERMONKEY, IRATEMONK, IRONCHEF, ISLANDTRANSPORT, JETPLOW, JUNIORMINT, KONGUR, LOUDAUTO, MAESTRO-II, MIDDLEMAN, MOCCASIN, MONKEYCALENDAR, NEBULA, NIGHTSTAND, NIGHTWATCH, OLYMPUS, OLYMPUSFIRE, OMNIGAT, ONTARGET, PHOTOANGLO, PICASSO, PROTOSS, QFIRE, QUANTUM, QUANTUMBISCUIT, QUANTUMBOT, QUANTUMCOPPER, QUANTUMINSERT, QUANTUMNATION, QUANTUMSKY, QUANTUMTHEORY, RAGEMASTER, RETURNSPRING, ROCKYKNOB, SCHOOLMONTANA, SEAGULLFARO, SEASONEDMOTH, SERUM, SIERRAMONTANA, SLICKERVICAR, SOMBERKNAVE, SOUFFLETROUGH, SPARROW II, SPECULATION, STRAITBIZARRE, STRIKEZONE, STRONGMITE, STUCCOMONTANA, SURLEYSPAWN, SURPLUSHANGER, SUTURESAILOR, SWAP, TARGETPROFILER, TAWDRYYARD, TOTECHASER, TOTEGHOSTLY, TRINITY, TUNINGFORK, TURBINE, TURBOPANDA, TURBULENCE, TURMOIL, TUTELAGE, TWISTEDKILT, TYPHON HX, UNITEDRAKE, VAGRANT, VALIDATOR, VIEWPLATE, WAGONBED, WATERWITCH, WISTFULTOLL, YELLOWPIN, ZESTYLEAK

A-Team • January 6, 2014 7:19 AM

Please don’t present acronym guesses as knowledge — they’re often been explained within earlier NSA documents:

CNO computer network operations
.. CNE computer network exploit [not exfiltration]
.. CND computer network defense
.. CNA computer network attack

NSA’s organizational chart took a big step forward with whole new divisions of TAO (S32) being revealed. We might well wonder why none of these had surfaced during the previous 7 months of disclosures (journalistic self-censorship? new document source?) Too bad, that gave them 7 more months of surveillance.

S32 221 DEITYBOUNCE
S32 221 GINSU
S32 221 GODSURGE
S32 221 IRATEMONK
S32 221 IRONCHEF
S32 221 SWAP
S32 221 WISTFULTOLL

S32 222 DROPOUTJEEP
S32 222 FEEDTROUGH
S32 222 GOPHERSET
S32 222 GOURMETTROUGH
S32 222 HALLUXWATER
S32 222 HEADWATER
S32 222 JETPLOW
S32 222 MONKEYCALENDAR
S32 222 SCHOOLMONTANA
S32 222 SIERRAMONTANA
S32 222 SOUFFLETROUGH
S32 222 STUCCOMONTANA
S32 222 TOTECHASER
S32 222 TOTEGHOSTLY 2.0

S32 23 COTTONMOUTH-I
S32 23 COTTONMOUTH-II
S32 23 COTTONMOUTH-III
S32 23 CROSSBEAM
S32 23 FIREWALK
S32 23 HOWLERMONKEY
S32 23 JUNIORMINT
S32 23 MAESTRO-II
S32 23 SOMBERKNAVE
S32 23 TRINITY

S32 242 CANDYGRAM
S32 242 CYCLONE Hx9
S32 242 EBSR
S32 242 ENTOURAGE
S32 242 GENESIS
S32 242 NEBULA
S32 242 NIGHTSTAND
S32 242 PICASSO
S32 242 SPARROW II
S32 242 TYPHON HX
S32 242 WATERWITCH

S32 243 CTX4000
S32 243 LOUDAUTO
S32 243 NIGHTWATCH
S32 243 PHOTOANGLO
S32 243 RAGEMASTER
S32 243 SURLEYSPAWN
S32 243 TAWDRYYARD

S32 3 DNT Data Network Technologies
S32 7 RTD Requirements and Tasking Development

A-Team • January 6, 2014 7:45 AM

Get your document issuance dates right. I’m see two sources for mistakes here — careless skimming of content and the confusing lower right hand corner CSSM 1-52 20070108.

That’s just the date of the issuance of the Communications Systems Support Manual 20070108 that the NSA document at hand is complying with. It’s on many released documents.

The most recent date in this document set occurs in the Booz Allen ppt two slides after the iPad hack — 01 April 2013. That dovetails rather well with Snowden’s last days of employment there.

The FoxAcid implant the back door on the iPad was successful after 21 days of trying — on 19 Feb 2013.

So many commentators have mixed up this stage1 VALIDATOR man on the side insert with DROPOUTJEEP — we know from earlier disclosures NSA has 3-4 distinct iOS attacks (including corrupt apps and synching with desktop).

Using Tera-WURFL, you can verify that the User Agent string in the iPad is valid — this training ppt walks new R&T analysts through an actual attack, not a hypothetical. iOS 5 is still in widespread use.

User Agent data: Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit 534.46 (KHTML like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3

Steven C. • January 7, 2014 6:59 AM

The 44x24mm hard plastic pad on my DL 380 G5 backplane board, is merely an insulating pad with soft foam beneath, glued over the Molex connector’s through-hole pins. But it could very discretely be swapped out for an IC and have electrical contact with the pins beneath.

Bernd • January 7, 2014 8:08 AM

I got access to several actively operational HP Proliant 380DL G5 edition servers. I just swaped out one unit and disassembled it to see if i can spot anything out of the ordinary. Unfortunately i did not find anything suspicious. There are serveral external I2C controller attached to pin headers but “they look legit”. I also collected a great deal of images from eBay and could not spot a single difference in between the systems. It’s like looking for a needle in a haystack. I’ll run i2cdetect now and compare this between two or more systems. HiRes Images and i2c data is available if it is of interest just post here, i’ll check back tomorrow.

Also i would be glad for any hint. Everyone is pretty much now in doubt now about our servers. They are already old legacy hardware but it’s obviously still a subject of interest.

-Bernd

goldfish • January 7, 2014 3:35 PM

1st point
This equipment is used to breech an air gap, hence the RF and short range. Many of us have our secure (thought but not) off network server sitting beside our public web servers in the room. (ok so the racks 20′ away)

Just need one on each side and the secure is not so secure. (ouch)

2nd) The risk to intercept also shows these are for high(ish) value targets.

Wonder who would use a ton of HP servers they illegally acquired? (With help they didn’t expect)

Nice tool, bet it works on many different servers.

Iain Moffat • January 7, 2014 3:54 PM

@Bernd and Steven C – Thanks for checking DL385 and 380 machines for us. I would be interested to see Bernd’s I2Cdetect result for the 380 G5 (although I believe it will be the same as the 385 with nothing directly visible)

Zakarro • June 14, 2014 2:40 PM

Hi everyone, I stumbeled upon this site while trying to diagnose what was wrong with the two rigs on my network.

I will explain, for over a year I was noticing weird crashes (CTDs) with games and apps, like AV crashing and mouse driver crashing, all with same exception code cx00005 which seemed to equate to a buffer overflow. At first I thought something was faulty with hardware mainly RAM, but I swapped RAM 3 times with same results. I ended up swapping EVERY part in rig, still same problem. I finally suspected being infected with a firmware rootkit, since I had already done internal secure erase wipes on my HDDs to rule out a regular malware infection on disks. Still same problem. So I definatly think I had some sort of firmware rootkit the question is what? You say ironchef needs to be implanted physically?

I suspect how that could have happened, If I recall correctly I remember selling a previous motherboard of mine on ebay, to my suprise the buyer returned it and gave us a negative review with one of the excuses being “it is broken and doesnt work” of course I tested the board when it was shipped back and it worked fine just like I sent it out. My theory is that what if the buyer was a perpetrator and infected the board knowing I would test it when I got it back and it somehow infected other parts I had connected to it? Im using a different board now but what if its infected thanks to the old board? I know it sounds far fetched but I cant posibly think of any other ways.

Im not a n00b I know what to click on and not to click on online, I run both hardware SPI and NAT firewall on router and very good software firewall. If I was indeed infected how could it have happened? Do you think it could have been the ebay buyer?

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.