HP Shared ArcSight Source Code with Russians

Reuters is reporting that HP Enterprise gave the Russians a copy of the ArcSight source code.

The article highlights that ArcSight is used by the Pentagon to protect classified networks, but the security risks are much broader. Any weaknesses the Russians discover could be used against any ArcSight customer.

What is HP Enterprise thinking? Near as I can tell, they only gave it away because the Russians asked nicely.

Supply chain security is very difficult. The article says that Russia demands source code because it's worried about supply chain security: "One reason Russia requests the reviews before allowing sales to government agencies and state-run companies is to ensure that U.S. intelligence services have not placed spy tools in the software." That's a reasonable thing to worry about, considering what we know about NSA's interdiction of commercial hardware and software products. But how can Group A convince Group B of the integrity and security of hardware/software without putting itself at risk from Group B?

This is one of the areas where open-source software has a security edge. If everyone has access to the source code -- and security doesn't depend on its secrecy -- then there's no advantage in getting a copy. As long as companies rely on obscurity for their security, these sorts of attacks are possible and profitable.

I wonder what sorts of assurances HP Enterprise gave its customers that it would secure its source code, and if any of those customers have negligence options against HP Enterprise.

News articles.

EDITED TO ADD (10/5): Commentary.

Posted on October 4, 2017 at 8:08 AM • 38 Comments

Comments

jonesOctober 4, 2017 8:24 AM

Sounds like we need an investigation in congress, a committee to review security measures, border searches, more defense contractors, and more globalized capital.

IanOctober 4, 2017 8:43 AM

Well! If it's good tight, secure code, then there's no harm done. Open source security and all that. However, if it's sloppy, security by obscurity type stuff, then HP deserves any opprobrium that comes it's way. As does the Pentagon for using it without due diligence.
Open the code up for public examination to see which way the penny falls!

JustinOctober 4, 2017 8:54 AM

I believe it is important to get the facts correct. HP Inc. and Hewlett Packard Enterprise are two separate companies. HPE made the decision however the inaccurate reporting of many sources has conflated it to be 'HP' this is incorrect.

Additionally, HPE split off the software assets and made this decision which they knew would not affect them or their business. It now affects Micro Focus who were unaware during the transaction period that HPE executive management made this decision.

Bruce SchneierOctober 4, 2017 8:55 AM

@Ian

Well, that's the issue of course. If the code is designed to be open, and open to everyone, then no one has an advantage. Russia can assure itself that the NSA hasn't put back doors into the code, and the US can assure itself that the Russians don't have any attack advantages -- and ArcSight customers can assure themselves that HP isn't selling their security down the river.

This is harder with hardware, but I think we need to move to open source for critical infrastructure.

keinerOctober 4, 2017 9:05 AM

@BS

"This is harder with hardware, but I think we need to move to open source for critical infrastructure."

Open source BIOS? Open source processors, mobos, graphics cards, HDDs, SSDs, monitors, keyboards, mice, cables?

Cool! Where do you wanna buy all that stuff?

Ross SniderOctober 4, 2017 9:28 AM

It's pretty typical for other governments, Russia included, to get the source code to software they depend on like full Microsoft Windows SC. (The Pentagon also depends on this code.

Is the issue here that Russia does not itself rely on ArcSight, but it got the source code to analyze anyway?

If this is the case, my guess is that the open vendor relationship just allowed the request to get processed in due course as a routine aspect of running the business. This is the same kind of open relationship that companies have with intelligence agencies and governments around the world: it's good business and its good partnership. So unless there's really effectively policies and guidance around the specifics, the relationship can extend further than it does theoretically on paper.

Who?October 4, 2017 9:29 AM

@ Bruce

How can customers assure that ArcSight has no backdoors? I am talking about ArcSight itself, not the source code. How can they assure their binaries are not backdoored? In other words, that binaries are being built from the same code they are auditing.

An unrelated example, OpenBSD snapshots usually contain experimental code that has not been committed [yet] to the development tree.

Who?October 4, 2017 9:35 AM

@ Bruce Schneier

You are right about moving to open source for critical infrastructure and, in general, any development that cares about security. It is the only way to go. Until then, we will have half-baked security solutions no one can really trust.

MikeAOctober 4, 2017 10:13 AM

Tracking "Did this binary come from an honest compilation of the source I have?" is a very tough problem, as evidenced by the (still in progress) work on "reproducible builds" among some OS distributions. I cannot believe it is helped by software suppliers who write code that can only be compiled by _one_ compiler, or by the twisty web of dependencies on a rat's-nest of dynamic libraries and "race to the top of the bat" search paths for executables and said libraries.

Not to mention that every time some obscure and low impact vulnerability is publicized, there is a chorus of "This is why we _MUST_ always enable mandatory automatic updates!"

"Know Your Customer" should be accompanied by "Know Your Vendor". And "Trust, but Verify (if you somehow can)"

Gunter KönigsmannOctober 4, 2017 10:56 AM

If the US wants to see the code that's legit. Even for me as an European and even if I know that they have a history of declaring backdoors as NOBUS instead of closing them. As an European I want my government to be allowed to see the source, too, because I fear NSA backdors. And I expect Chinese and Russian citizens to feel the same => what's the problem? As a side note nobody even knows if the code shown to the government has the same backdoors as the Real Thing, if it has backdoors, that is.

Clive RobinsonOctober 4, 2017 11:04 AM

@ Bruce,

What we HP Enterprise thinking

You might have "over adjusted" when responding to @Justin's comment.

Clive RobinsonOctober 4, 2017 11:15 AM

As far as I remember China has banned the imoport of all foreign ICT unless they have all the software etc...

There are three ways you can look at this,

1, Chinese National Security.
2, Chinese technology theft.
3, Chinese cyber-weapon development.

The real concern of many businesses is "Chinese technology theft" which has been going on one way or another for quite a while now.

The problem with "Open Source" for making such products is in the short term "technology theft". Especially when many costs of manufacture are considerably less in places like China.

One solution to this would be to make the technology multiple level, thus only the basic levels are "Open Source" whilst the more desirable parts remain closed source. It's then upto the purchaser as to which option they go for.

BobOctober 4, 2017 11:35 AM

It's commonplace for a 3rd party to do an audit of source code before a purchase. Allowing the Russians direct access to the source code, knowing it is used at the Pentagon, that warrants a serious investigation.

China does require access to source code, that is why so many vendors try to keep their dealings with them secret.

Impartial observer classOctober 4, 2017 11:47 AM

"Well! If it's good tight, secure code, then there's no harm done"

It's HP code...

Here's a question, why are they ALLOWED to sell a SECURE platform used by the Pentagon to effing Russia or anyone else?

If you're going to get on the big MIC gravy train you shouldn't be allowed to jump off at any point you want for a quick side-profit from ADVERSARY NATION STATES.

What is this, Halliburton?

Whoever decided to use HP SW in the first place... you know what, no. I'm not going to allow the idiocy to get to me today, I'm going to pretend everything is perfectly fine.

When the power goes off for 3-4 months they'll pay more attention to important stuff.
No use worrying until it happens, there is zero impetus in Congress to fix this now.

People literally have to die before anything will even MAYBE get done around here.

swashbucklerOctober 4, 2017 12:02 PM

I'd be careful about saying they "gave' the Russians the source code. I used to work for a company that did something similar and the design of the inspection was such that we never intentionally allowed them to copy the source. It's been too long ago that I don't remember all of the details, but I do remember that we thought with what we setup they'd have to go to some pretty extreme measures to get a copy of the source (e.g. hidden cameras in glasses and then page through literally hundreds of MBs of source, that sort of thing).

Also, at least officially, the reviewers were not members of a "Russian defense agency", but a private company - though with the Russians who knows. The reviewers were all youngish, late 20s, early 30s, at least at my location.

SofaOctober 4, 2017 12:22 PM

Bruce,

Minor typos:
You have:
What we HP Enterprise thinking? Near a I can tell, they only gave it away because the Russians asked nicely.

Should read:
What were HP Enterprise thinking? Near as I can tell, they only gave it away because the Russians asked nicely.

You have:
The article says that Russia demands source code because its worried about supply chain security

Should read:
The article says that Russia demands source code because it's (added apostrophe not for possessive but for 'it is') worried about supply chain security

or
The article says that Russia demanded the source code because it was worried about supply chain security

You scratch my back, I hack you later.October 4, 2017 2:23 PM

Rather than correct typos, read this :

https://theintercept.com/2017/03/10/government-zero-days-7-years/

Check out that graph of "living" and "immortal" 0 days. Average 7 year(!) lifespan per.

Yeah sure let's rent out secret crufty closed source code all around the place.

I'll tell you exactly what happened. HP didn't just send it along for free vodka.

Russian company says "We need the source code, or we don't buy your crap, get it?"

HPE : Tosses them a thumb drive. "Backrub?"

trentOctober 4, 2017 4:43 PM

But ... we heard about it? So it's somewhat more transparent than the American system where a vendor wouldn't even be permitted to announce that it received a National Security Letter.

Also, from the article, "no one Reuters spoke with was aware of any hacks or cyber espionage that were made possible by the review process" - ahahah of course not, that's why they're called zero-days.

Further: "HPE agreed last year to sell ArcSight and other security products to British tech company Micro Focus International Plc in a transaction that was completed in September" - the UK is in theory friendlier to the US than Russia (and it wasn't Micro Focus who shared the code) but that cat sounds like it's well out of the bag.

PeanutsOctober 4, 2017 6:30 PM

HPE, The new Kaspersky, who customers can now anticipate RCE’s which would have previously out of reach.
Nice HPE...

Nice gift to Splunk salesmen everywhere
WTG

supersaurusOctober 4, 2017 6:51 PM

@Who?

How can customers assure that ArcSight has no backdoors? I am talking about ArcSight itself, not the source code. How can they assure their binaries are not backdoored? In other words, that binaries are being built from the same code they are auditing.

as you say, having the source code doesn't prove anything about the binaries. even if you have their build system and the source code for that good luck figuring it out. remember the famous ken thompson unix password hack? http://wiki.c2.com/?TheKenThompsonHack. nice proof of concept.

Ken describes how he injected a virus into a compiler. Not only did his compiler know it was compiling the login function and inject a backdoor, but it also knew when it was compiling itself and injected the backdoor generator into the compiler it was creating. The source code for the compiler thereafter contains no evidence of either virus.

Clive RobinsonOctober 4, 2017 8:30 PM

@ Who?,

How can customers assure that ArcSight has no backdoors?

They can not, not even with all the source and the same compiler setup etc.

Have a think back to the NSA bludgeing NIST with Dual EC DRBG. Arguably that was the first "NOBUS" that was caught from that $250million/year BULLRUN budget.

Which means that there are probably other NOBUS from the NSA and other SigInt agencies out there, that we do not yet recognise as such...

Have a read up on Cryptovirology and Kleptography from Adam Young and Moti Yung. Moti has quite an interesting CV the highlights of which are up on Wiki.

How can they assure their binaries are not backdoored? In other words, that binaries are being built from the same code they are auditing.

This is an entirely different question. As many embedded hardware engineers will tell you the high level source code "is nice" but at the metal you work with binary code.

When embedded engineers develop systems they need "test tools" over and above a voltmeter and oscilloscope. These can be expence In Circuit Emulators (ICE) jtag analysers or logic analysers. All of which have available binary to assembler nemonic converters.

Thus the engineers become fluent in the low level assembler coding. Which to some are actually way better than the original high level language source code.

If you have both tying them together is usually not that difficult, just tedious.

So if the ArcSight could be built and debugged, then another engineer from a diferent organisation can pull it apart and "debug it".

Bryce AustinOctober 4, 2017 8:56 PM

You bring up a very good point on how to level the playing field Bruce. Is HPE's best course of action now to make the ArcSight source code publicly available?

MeOctober 4, 2017 9:06 PM

Before everyone starts grabbing pitch forks and going after HPE. You need to understand what Arcsight is.

First is protects nothing. Really not joking. It is a central place to dump all you security logs from different platforms and apps. It gives you IA cyber people a way to run queries to look for distributed attacks.

If they don't look, and don't know what the log entires mean, than it doesn't even do that much.

I see it all the time. The people in the gov that have Arcsight have no idea what they are looking at.

So mainly the app listens on syslog ports and few others, dumps stuff into a DB of choice, and has a web frontend. That shouldn't be too hard to protect.

mostly harmfulOctober 4, 2017 9:47 PM

@Bruce Schneier writes [emphasis mine]:

Reuters is reporting that HP Enterprise gave the Russians a copy of the ArcSight source code.

@swashbuckler is doubtful that HPE did any such thing.

And indeed the Reuters article reports, in so many words, that HPE says they did not allow code to leave their premises [emphasis mine]:

An HPE spokeswoman said source code reviews are conducted by the Russian testing company at an HPE research and development center outside of Russia, where the software maker closely supervises the process. No code is allowed to leave the premises, and HPE has allowed such reviews in Russia for years, she said.

OtterOctober 4, 2017 10:42 PM

Any competent mil/gov which demands to see the source code is not hoping to rip it off. They can decompile it, like any teenager lurking in her father's basement.

In anycase, it is cheaper to let HPEwhoever build and maintain it, than to do so themselves, cheaper than to maintain decompiled code, even cheaper than to maintain stolen code. Like any customer, including our pallid teenager, they would like assurance that the code quality is grade-A, and that some critical (to them) parts do exactly what they want. Some customers are more privileged than others.

Also, mil/gov - competent remember - is haunted by the ghosts of trojans and zero-days past. They do not expect see the label "NSA-Back-Door". Nor do they expect proof that this object code was produced from that source code. They will send some really sharp programmers and analysts to look for crappy, obfuscated, and insecure code. To be sure, places where an incompetent vendor might try to hide "NSA". But really, places where a competent vendor might miscalculate a date. They will send a few architecture types with excellent memories for the structure of the product, where stuff is, where stuff happens, where undocumented features might be inserted. Where documented features can requested or removed. Where the light switches are or can be installed in the black-boxes. Where the hardware might need bolstering, where it will confuse the clerics, etc.

Finally, being haunted as well as competent, when they get a copy home, they will turn it over to the decompilers, the security analysts, a teenager, and a room full of open source monitoring software. Those people who examined the code will tell them the first places to look and what should be found there.

But first of all, if it is very critical, they will code it themselves. Not trust but verify. Distrust and also verify. There are not many innocents left.

HermanOctober 4, 2017 11:19 PM

I don't see a problem with that.

Anyone can peruse github, kernel.org, savannah, freebsd and more. Also, MS makes the Windows source available to other countries, including Russia and China.

So, paranoid much?

Clive RobinsonOctober 5, 2017 1:19 AM

@ Otter,

Any competent mil/gov which demands to see the source code is not hoping to rip it off.

Thus during the first stages of looking at the product after picking up a brochure at a show etc, they would find,

https://blogs.wsj.com/deals/2010/09/13/who-is-a-big-fan-of-arcsight-the-cia/

Which indicates ArcSight is in business in part due to the CIA "venture fund",

    In-Q-Tel, which is staffed and funded by the CIA, provides its portfolio companies not just with money, but with intelligence and counter-intelligence expertise and a “test-bed” for new products.

Which actually might be seen as a vote of confidence in the ArcSight products, all things considered.

The Russian's have a saying about "rouges" and being close to them, which is not to disimilar to the old English saying "When you sup with the devil, use a long spoon".

So yes they might want to check the horses teeth as it were ;-)

Clive RobinsonOctober 5, 2017 3:03 AM

There is a flip side to this argument that ought to be high lighted which is,

If US National Security is of such importance to the USG, why does it buy "Consumer Off The Shelf" (COTS) software and not develop it's own?

It's primarily an economic question, but there is a secondary consideration arising from it which is,

Does the USG have the right to block what is an independent commercial software company from seeking sales abroad? Simply because the USG uses the software.

It's an issue that I expect will come up more in the future, and people will point at the Export Controls for Conventional Arms and Dual-Use Goods and Technologies, or as it's more commonly called "The Wassenaar Arrangement" (WA)[1].

The WA was designed to control the sale and trafficking of conventional weapons and dual use technologies, which can have both a civilian and military purpose. Or to put it another way try to ensure you have the upper hand / leading edge on potential hostile nations.

It has come under fire in the past[2] due to attempts to vastly increase it's scope, in ways that are too broad to be considered safe. Prior to that it had a staring role in the first of the Crypto-wars, where it was mocked by crypto geek tee shirts.

The problem is ArcSight is in essence a distributed database report generator. That is it takes input log files from any device in a enterprise ICT environment that the security bods decide should be included. It then performs an analysis on those files and makes a report through a near real time interface.

If you decide ArcSight to be a "munition" or "Dual use" technology then that will spill over into all other database and report generators, which would frankly make it a daft thing to do (not that being daft is something Politicians are strangers to ;-)

The real problem is the Politicos want it both ways. That is they want the techbology at low comnercial prices, but also they want the sort of control only eye wateringly bespoke defense R&D projects give. Realistically they can not have it both ways as you need a market place with sufficient demand not just for a product to be thought of but actually built in the quantaties that will bring the price down.

Any way for those in London say hello to another grey, dreary, pouring with rain morning commute to "the coal face", it actually makes this stuff look interesting in comparison ;-) Time for a nice cup of tea then hit the keyboard and arrange some meetings...


[1] The Wassenarr Arrangnent --not to be confused with the agreement-- is named after a town and it's environs in south Holland. Where the arrangnent was first put to paper in july 1996, as an international arms-control agreement. It is a "MECR" or Multilateral Export Control Regime, that replaced the more stringent "Cold War" "Coordinating Committee for Multilateral Export Controls" (COCOM). Currently the WA has 41 participating states/nations including the US and many of West and East European countries including a number of former Warsaw Pact nations.

[2] https://www.wired.com/2015/06/arms-control-pact-security-experts-arms/

trentOctober 5, 2017 4:57 AM

@Clive Robinson quoted:

> Does the USG have the right to block what is an independent commercial software company from seeking sales abroad? Simply because the USG uses the software.

Haven't record companies spent decades including exclusivity clauses in artist deals? Commercial exclusivity is a solved problem.

You put this in your RFP up front, the vendors say "oh, we don't have foreign sales? let me adjust the cost to make up for that" and you're good. Yes it's more expensive, that's kind of the point.

> Realistically they can not have it both ways

Because the other question is: can a vendor afford to continue to exist if they're only getting domestic money from a product they budgeted for being able to sell internationally?

TatütataOctober 5, 2017 9:13 AM

Before everyone starts grabbing pitch forks and going after HPE. You need to understand what Arcsight is.

First is protects nothing. Really not joking. It is a central place to dump all you security logs from different platforms and apps. It gives you IA cyber people a way to run queries to look for distributed attacks.

If this is apparently such a trivial (?) piece of code, why should the Rooskies buy it from the US and make a fuss about the source code, instead of developing it themselves? Dito if it is instead a national-security relevant tool.

Your description ArcSight sounds somewhat like the general promise of "PROMIS", which was, as much as I can deduce from the scant details provided, a kind mainframe-era database for organising and correlating case and suspect information.

This system was sold to a number of police forces, including the Canadian RCMP. The nag-smelling Mounties were apparently provided a backdoored version. The story petered out into a full tin-foil hat conspiracy theory with all the ingredients (FBI, Mossad, murders, etc.), so it's hard to make out what may have actually happened.

You might get slightly wiser from a recent FOIA-based Muckrock story:

The court ruled in Inslaw’s favor, finding that the DOJ’s officials “took, converted and stole” the [PROMIS] software through “trickery, fraud and deceit.”

JanOctober 5, 2017 11:24 AM

@keiner


Open source BIOS? Open source processors, mobos, graphics cards, HDDs, SSDs, monitors, keyboards, mice, cables?
Cool! Where do you wanna buy all that stuff?

BIOS/firmware (see "How to get hardware with coreboot")

Processor and board (there are other RISC-V boards)

Searching for "open source keyboard" shows several designs.

Clive RobinsonOctober 5, 2017 12:56 PM

@ Jan, Kiener,

Processor and board (there are other RISC-V boards)

Funny you should say that, I posted about a high end quad core 64bit version just a day or so ago,

https://www.schneier.com/blog/archives/2017/09/friday_squid_bl_594.html#c6761631

The problem with the hardware from my point of view is "Even if it is open source, you can not realisticaly edit it yourself" thus all sorts of Intel ME ring -3 or worse could be tucked away inside the chip just waiting to hatch out.

After all if RSA "trousered" 10million of NSA BULLRUN money just to make Dual EC-DRBG the default Random Number Generator in RSA's suite of programs, ask yourself how much cash it would take to buy out the loyalty to customers of a small strugling fab?

Now we're making things up!October 5, 2017 2:27 PM

" like any teenager lurking in her father's basement. "

What is this analogy!?

JanOctober 5, 2017 3:10 PM

@Clive Robinson

The problem with the hardware from my point of view is "Even if it is open source, you can not realisticaly edit it yourself" thus all sorts of Intel ME ring -3 or worse could be tucked away inside the chip just waiting to hatch out.

The Intel ME would certainly be detectable given source code, an open toolchain, and sufficiently powerful magnification (which I'd guess is getting somewhat cheaper these days). Doping attacks are said to be undetectable by contrast, but seem impractical for FPGAs (when you don't know where people will put the "important" gates). Maybe we can use that to our advantage: give enough reconfigurability for us to "randomize" the hardware or run thorough tests on individual units. Too bad none of the powerful open-source CPUs can be built with an open-source FPGA toolchain.

Clive RobinsonOctober 5, 2017 10:37 PM

@ Jan,

Too bad none of the powerful open-source CPUs can be built with an open-source FPGA toolchain.

Yes, as I understand it theres a lot of IP involved and in some cases it's seen as the competative edge.

But if you want "powerful" FPGA is not the way you would go if you could avoid it. Because although it will give a boost over software it's still not close to naked silicon[1].

Which is the problem you have, to use somebody elses chip effectively "sight unseen". Even if you personally could get to see the individual logic components, there is a lot of custom work involved so working your way through a design looking for just a few subtal "tricks" might be close to impossible in the length of time the actual part is available to buy and use in a product.

Further it's probably not you who is going to do the job of looking, it's highly specialised and such people are a bit 'like hens teeth, you hear about them but you never see them'. Thus you get a trust issue, it's the same for anything you can not do personally.

I could go one about many of the stages and trust issues --we have here in the past-- but by no means all are known by me or most people who read this blog, due to the highly specialised nature of the work.

It's why I decided that trying to lock down all the parts of the supply chain were not feasable for the many. After all as we know even the NSA with it's culture, clout and funding can not hang on to things[2]. Thus it would be daft to assume people could not be got at one way or another[3].

There is an old saying about 'you cann't prove a negative', in general it's true. So I just accepted that like Intel "they would be inside" as I could not prove otherwise...

So having realised it was not only impractical to design a secure system the traditional way, but actually impossible in human terms, that there must be other ways, that had to be thought of and investigated.

The thought that occured is that "all bubbling up attacks" have to be more or less unique to each architecture to be of any real use to an attacker. Likewise any tricks would have to get through unknown testing without raising flags. Without going through the whole reasoning tree you can see that the attackers need for stealth/secrecy and effectiveness "boxes them in" and you can use that against them.

I realised that even if they are inside you can mitigate against them by a number of methods (which likewise have been discussed here before).

So whilst "Open Source" is nice, it's by no means essential when it comes to the hardware. Thus the question moves to "finding the design sweet spot".

[1] I gather some CPU manufacturers are currently looking at including FPGA within their CPU designs to make what is in effect a co-processing element to improve algorithm speed between five and twenty five times. Whether we will see them at the commodity pricing point appears unlikely from what has been said about it so far.

[2] It appears that yet another "take it home to work on" person has been found at the NSA. And attack tools or similar have escaped / been liberated, potentially to a foreign power. It's all "undisclosed sources" so fact/fiction issues arise, so I'd be speculating like some of those "sources" if I said anything more. But let me say it would not surprise me the way things work these days.

[3] There is an acronym in the IC of "MICE", it's basic assumption is everybody has buttons, you just have to push hard enough, to get them to do what you want. And if they don't, well just shove them under a bus and work on the next body in line, you'll find one eventually.

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of IBM Resilient.