NSA Running a Massive IDS on the Internet Backbone

The latest story from the Snowden documents, co-published by the New York Times and ProPublica, shows that the NSA is operating a signature-based intrusion detection system on the Internet backbone:

In mid-2012, Justice Department lawyers wrote two secret memos permitting the spy agency to begin hunting on Internet cables, without a warrant and on American soil, for data linked to computer intrusions originating abroad—including traffic that flows to suspicious Internet addresses or contains malware, the documents show.

The Justice Department allowed the agency to monitor only addresses and “cybersignatures” ­—patterns associated with computer intrusions—that it could tie to foreign governments. But the documents also note that the N.S.A. sought to target hackers even when it could not establish any links to foreign powers.

To me, the big deal here is 1) the NSA is doing this without a warrant, and 2) that the policy change happened in secret, without any public policy debate.

The effort is the latest known expansion of the N.S.A.’s warrantless surveillance program, which allows the government to intercept Americans’ cross-border communications if the target is a foreigner abroad. While the N.S.A. has long searched for specific email addresses and phone numbers of foreign intelligence targets, the Obama administration three years ago started allowing the agency to search its communications streams for less-identifying Internet protocol addresses or strings of harmful computer code.

[…]

To carry out the orders, the F.B.I. negotiated in 2012 to use the N.S.A.’s system for monitoring Internet traffic crossing “chokepoints operated by U.S. providers through which international communications enter and leave the United States,” according to a 2012 N.S.A. document. The N.S.A. would send the intercepted traffic to the bureau’s “cyberdata repository” in Quantico, Virginia.

Ninety pages of NSA documents accompany the article. Here is a single OCRed PDF of them all.

Jonathan Mayer was consulted on the article. He gives more details on his blog, which I recommend you all read.

In my view, the key takeaway is this: for over a decade, there has been a public policy debate about what role the NSA should play in domestic cybersecurity. The debate has largely presupposed that the NSA’s domestic authority is narrowly circumscribed, and that DHS and DOJ play a far greater role. Today, we learn that assumption is incorrect. The NSA already asserts broad domestic cybersecurity powers. Recognizing the scope of the NSA’s authority is particularly critical for pending legislation.

This is especially important for pending information sharing legislation, which Mayer explains.

The other big news is that ProPublica’s Julia Angwin is working with Laura Poitras on the Snowden documents. I expect that this isn’t the last artcile we’re going to see.

EDITED TO ADD: Others are writing about these documents. Shane Harris explains how the NSA and FBI are working together on Internet surveillance. Benjamin Wittes says that the story is wrong, that “combating overseas cybersecurity threats from foreign governments” is exactly what the NSA is supposed to be doing, and that they don’t need a warrant for any of that. And Marcy Wheeler points out that she has been saying for years that the NSA has been using Section 702 to justify Internet surveillance.

EDITED TO ADD (6/5): Charlie Savage responds to Ben Wittes.

Tags: , , , , , , , ,

Posted on June 5, 2015 at 7:42 AM • 45 Comments

Comments

CJD • June 5, 2015 8:10 AM

Seems legit, I mean, they were only collecting phonecall data on calls originating / terminating in foreign nations, and that went over just fine, right?

I love how the other articles on this I have read try to cast it in the light of protecting us, and how they are just keeping the bad guys out. If that was the real case, A) they are failing pretty miserably, and B) I’m not sure anymore who is a bigger threat, the “bad guys” or the NSA.

AlanS • June 5, 2015 8:49 AM

Also see Marcy Wheeler’s Section 702 Used for Cybersecurity: You Read It Here First which has links to numerous discussions of this issue on EmptyWheel going back to June 2013. She comments:

I have reasons to doubt that the NSA documents released tell all of the story, as I hope to show in upcoming posts

For a rather different take on all this see Wittes’ rubbishing of the Charlie Savage’s NYT story Et Tu, Charlie? The New York Times’s Savage NSA Blunder. He also promises his own analysis of the documents in coming days.

keiner • June 5, 2015 8:50 AM

@MAce M

Industry espionage? As it is in Europe, with the NSA selectors provided to the German secret service (BND). Selectors included eg.

  • Eurocopter
  • EADS
  • Siemens
  • and several (hundred?)thousand others
  • Several federal agencies in Europe

As the BND and NSA do not want to hand over these selsctors to the parlament commission entitled to control the BND, this commission has stoped to allow new BND operations. Until they hand over the list to the parliament. Democracy at its best. But I personally the NSA/BND will win even this game….

Stein • June 5, 2015 9:01 AM

“If they can’t even protect government agencies from the “largest thefts of government data ever seen” by a foreign country, what value is the program?”

It makes one wonder if all the information they’ve been secretly collecting might also get stolen …

Chelloveck • June 5, 2015 9:11 AM

I used to by cynical, now I’m just depressed. AFAICT it doesn’t matter what the law is. The NSA does what the NSA does regardless of legal mandate, because everyone with the power to stop them actually agrees with what they’re doing. They may bluster about privacy and freedom and put on a good political theater show for the plebes, but in the end they all like the idea of having access to all the information about everyone. If they can get people to stomach laws that let them pry, great. If not, they’ll just do it anyway while they say anything they think their constituents want to hear. I like to pretend that the majority of the politicians have the sincere but misguided belief that everything they’re doing is for the good of the country, because the reality of it is just too dismal.

Sam • June 5, 2015 9:26 AM

@Stein

if all the information they’ve been secretly collecting might also get stolen …

In short, Yes. That mechanism has been referred to here previously as Third Party Collection. It happens enough that there’s a term for it.

There is also a term – Fourth Party Collection – where agency A gathers some surveillance data, agency B hacks them and steals it (that’s the Third Party Collection), then agency C hacks B and steals from B the intel originally collected by agency A.

EZSmirkzz • June 5, 2015 9:26 AM

There they go again. (Couldn’t resist.) Obvious to me is the selectors of proxies and encryption. Either one of which will get the three letter boys attention, and should, given the nature of the American character, provide substantial Echelon Day type lulz for years to come, while preventing our national media elites from snooping on our very, very interesting communications and political observations. One cannot overlook the the overlap of computer networks and human networks when granting the government the power to snoop, or giving it any information whatsoever. What people in cities don’t know, people in small towns understand.

Of course everyone noticed the document dump was quickly followed by news of the government HR hack, a basic reminder that the purpose of the public face of the three letter agencies is lie to the public, something so obvious it was used as a line in MI-5, the British spook series.

Much of what we hear is security theater for public consumption just like the scanners at airports, which are more likely jobs programs in a guise that conservatives will support. Probably too great a leap to assume it could all be a devious Koch brothers plot to sell more toilet paper to the huddled American masses, but then again in a nation that has collectively lost its’ mind the public have to deal with probabilities and not possibilities, which according to the government, is the business of the government.

Bob S. • June 5, 2015 9:54 AM

So much for the military staying out of domestic criminal investigations.

Scratch Posse Comitatus off the list of protections from abuse of power by the military, which we have all suspected for a long time already. It also explains how naval investigators got involved in pedophilia investigations in the state of Washington, the entire state…They do it because they can. And there is no one to stop them.

Meanwhile, “The Justice Department allowed the agency to monitor only addresses and “cybersignatures”…” More crap from the spies making up their own rules, in secret, on the fly as they see fit. It’s like they are saying, “we don’t rob banks on Tuesdays because it would look bad”.

And, despite the massive lawlessness, over-reach, and flaunting the rule of law, they are no good at what they do. No good whatsoever at preventing cyber attacks as in just today’s example of the massive grab of federal personnel records.

Meanwhile, as long and widely predicted, Congress shows no inclination to do anything at all about it as evidenced by the ludicrous Freedom Act.

This might be funny if it weren’t so sick.

Was I too harsh?

Winter • June 5, 2015 10:19 AM

Again, there is universal surveillance and no privacy because that is needed to keep the USA safe, or so they say.

But all this might and money seems to have been utterly wasted because they fail miserably at what their core business is: Keep the USA safe.
(see latest breach)

Another angle to this is that the stated goals of the NSA et al are not to protect the government and people of the USA (in that order).

Just as there are super-top-secret surveillance programs, their might be super-top-secret goals.

So we can chose between:
The NSA et al are
1) incompetent
2) traitors

Daniel • June 5, 2015 10:39 AM

Unlike Bruce I continue to think the real issue is this one: Information about Americans sometimes gets swept up incidentally when foreigners are targeted, and prosecutors can use that information in criminal cases.

The problem isn’t that there is no longer a useful distinction between “spy” and “hacker”–,b>the problem is that the NSA and the FBI think that there is no longer a valid distinction between foreigner and American. . It is this truth that makes Wittes comment a POS.

Marcos El Malo • June 5, 2015 10:39 AM

@Winter

One really wonders about the intelligence community’s agenda.

Especially when one looks at the larger context.

• Huge wealth inequality
• Pending man made environmental disaster
• Militarizing of police

What is it that they are preparing? The Zombie Apocalypse?

Maybe it’s all a coincidence.

Anon123 • June 5, 2015 11:16 AM

when a data breach occurs on American soil, and the NSA intercepts stolen data about Americans, it believes it can use that data for intelligence purposes

So the NSA intercepts the data exfiltrated from the Office of Personnel Management by Chinese attackers, and now can use it as it sees fit. Potentially any part of that data my be used as future selectors by being part of a cyberattack. The NSA can then create selectors on the personal information of government employees and begin collecting wholly domestic communications, but it is not domestic surveillance it is a cyberattack investigation.

Tinfoil Hat Speculation Time:

The NSA feels it has the greatest technical and surveillance advantage of any intelligence agency around the world. In an effort to fulfill their “collect it all” directive, they have shown they are willing to undermine critical security infrastructure, presumeably because their technical advantage will leverage more value from those vulnerabilities than their adversaries. If the NSA were backed into a corner and had no other legal means, might they use the same rationale of being able to leverage more value from data than adversaries and help facilitate, or not interfere with, a cyberattack that exfiltrates large amounts of personal information on Americans to give them further legal justification to continue their surveillance. Or for that matter, why not collude with a Five Eyes partner to have them “steal” the information in such away that the NSA can intercept it as foreign exfiltration of data, and then use it to expand the legal justification of domestic surveillance.

LessThanObvious • June 5, 2015 11:51 AM

If they truly operate as an IDS system and not a true surveillance system this doesn’t actually bother me. The internet is a public network with global security threats that do need to be understood and guarded against. When I put IDS on a corporate network I don’t inform users or think of it as surveillance. Yes data is captured, but unless the data matches an attack signature there is no reason to view it. Automated intrusion prevention on the other hand can be a big issue because of potential for false positives and blocking legitimate traffic from NATed networks. No matter what they tell us I’m afraid the reality is that the government is going to watch traffic at the border. Until their definition of “collection” changes you’d best assume they have a large portion of domestic traffic stored that can be analyzed any time someone gives authorization. Overall there is overreach, but IDS alone isn’t what I think we have to fear.

anon777 • June 5, 2015 12:02 PM

NSA, why don’t we just protect Americans through more secure computing (e.g., encryption)? It’s effective, it doesn’t destroy civil liberties, and it can be effectively universal if you’d work with the tech industry instead of breaking their products and slapping gag orders on them.

Clive Robinson • June 5, 2015 12:17 PM

On first reading about the data loss I thought the usuall “yawn yawn so what” because we have been effectivly conditioned into thinking “it happens all the time”.

Then I started thinking a little more on the subject…

And the thought came to mind was “the NSA have the expertise, why did they allow it to happen?” Which naturally led to the follow the money question of “What’s in it for the NSA?”.

Normally at this point I would share my thoughts on this, but this time I’m going to hold back and let others think on the issue and post on it, as a couple of posters have already…

I’m curious to see if others thoughts are as dark if not darker on the issue.

>> • June 5, 2015 12:52 PM

@Less than obvious, there are IDSs and IDSs. Some we could run would get us prosecuted for felony offenses. No prizes for guessing which kind NSA favors. Time for us all to start running honeypots and crowdsource the privacy interference evidence that the government withholds.

https://www.honeynet.org/node/1177

Clive Robinson • June 5, 2015 12:52 PM

@ LessThanObvious,

The thing about an IDS is that the data it captures to check against “knowns” is generally transitory. In that you only keep data for a very short period of time.

The much stated goal of the NSA is to “Store everything” to build what is in effect a time machine.

Thus I suspect the NSA’s idea of what their IDS is and what your corporate connectd IDS is are radicaly different in many respects.

But there is a secondary issue, when you work for a company, you are working on their dime on equipment their penny paid for, this gives them certain rights which are usually in the employment contract or associated documents, and you normaly have to sign a document saying you have read and understood them. Thus there is a contract in place where the terms have to be lawfull and lawfully applied.

What the NSA et al is doing is monitoring the endeavors of citizens who they have no contract with, who they are not paying and importantly do not own or have lien on the equipment the citizens use. Further what they do is most probably not legal if the citizens would be granted “standing” to take it to court.

Thus the two situations are as Chalk and Cheese.

LessThanObvious • June 5, 2015 3:06 PM

@Clive Robinson, You have a point there it is a different situation. Corporate IDS is also generally focused on traffic coming from the internet to a network owned by the IDS operator. Since the government ethical compass is a bit skewed from what most of us would like I guess it really would be best if they only monitored government owned networks. Leave the protection of private networks and private industry in the hands of private networks and private industry. Like the Sony hacks, I really couldn’t understand why people seemed to think government should have any role in protecting them, they should have the ability to protect their own networks. If the private citizen or private company is attacked then they have the ability to involve the FBI or whatever government resource they choose and we don’t have tolerate bulk collection for that to happen.

Justin • June 5, 2015 3:51 PM

@ LessThanObvious, Clive Robinson

Ensuring the security of private networks and private industry is part of the role of the NSA.

“If the private citizen or private company is attacked then they have the ability to involve the FBI or whatever government resource they choose and we don’t have tolerate bulk collection for that to happen.”

Private citizens and private companies (unless they are very large companies) do not have the FBI or other government agencies in their pocket or at their beck and call. Furthermore, in the vast majority of cases, they either don’t know or don’t care that they are being attacked. And when hacking and intrusion attempts make up a large portion of internet traffic, it is perfectly appropriate to do something about it on the backbone level. We shouldn’t have to tolerate endless ssh login attempts, botnets, and the spam they send out, when there are ways to filter out some of that garbage on the backbone.

The trouble is that most people don’t protect their own networks, and they create a public nuisance. The government certainly does not have the resources to deal with individual cases of computer intrusion, without dealing with them in bulk.

AlanS • June 5, 2015 4:40 PM

@Clive

Failure is success for the intelligence services. Failure means they didn’t have enough powers, resources, etc. Nothing to do with incompetence! All those scary threats? Give them more money, more powers! And keep doing it every time they fail. They have a bright future ahead.

Clive Robinson • June 5, 2015 5:13 PM

@ AlanS,

Nothing to do with incompetence! All those scary threats? Give them more money, more powers! And keep doing it every time they fail.

Well if the FBI are not incompetent they obviously must be increadably inefficient, lazy or both, and can not be bothered to do actual investigative work, now they are so addicted to that usless fire hose of data that wizzes by,

http://www.washingtonpost.com/blogs/the-switch/wp/2015/06/04/fbi-official-companies-should-help-us-prevent-encryption-above-all-else/

What they are actualky asking for is “a suicide pact” that will mean the death of the US ICT sector with the attendent loss in GDP and more importantly export income. I’m realy suprised the US Treasury have not yet started jumping up and down at the “ugly sisters” of the NSA&FBI and the harm they are hell bent on causing the US balance of payments.

AlanS • June 5, 2015 5:26 PM

@Clive

Yes, I find it hilarious that when it comes to intelligence and security government, and secret, unaccountable government to boot, is all good. Just keep them away from the economy, health, education, etc. because everyone one knows they are completely incompetent in all those areas!

Clive Robinson • June 5, 2015 5:39 PM

@ Justin,

Private citizens and private companies (unless they are very large companies) do not have the FBI or other government agencies in their pocket or at their beck and call. Furthermore in the vast majority of cases, they either don’t know or don’t care that they are being attacked.

Nor do citizens or businesses have the FBI on call for crime and crime prevention, so those that do not take the precaution of locking doors etc take a big hit.

Whilst private citizens have a duty of care to themselves they also under common / tort law have a duty of care to others. Go and look up what “an attractive nuisance” is and what the remadies are. As any lawyer will point out to you no matter how impossible it is for an individual to know and understand every piece of legislation, ignorance of the law is no defence under law.

Where the US has gone badly wrong as I’ve indicated in the past is that they have idiotically called what is only “cyber-crime”, “cyber-warfare” and thus take entirely the wrong attitude to it.

Just to remind you the “misson statment” of the armed forces can be summed up as “Go to new places and meet new people and by overwhelming weaponry kill them.”. Contrast that with the ages old definition of the “watch” and later the police which is “To hold the Kings Peace.”. The former emphasizes “death and destruction” whilst the later emphasizes “peace”. The former is plain mindless stupidity the latter is a thoughtfull and progressive attitude to life. I don’t know about you but I’m rather more infavour of the latter not the former, because I would still like to be able tosleep at night without having to worry about having a drone strike turning me, my house or those around it into plasma and high kinetic energy debris.

Which is why I think cyber should be dealt with by local and international law and appropriate law enforcment, not a bunch of blood thirsty megalomaniacs with zero morals, we politly call “War Mongering Fools”.

Justin • June 5, 2015 6:21 PM

That headline from the Washington Post blog is a bit of a sensationalist misquotation.

Steinbach, according to the actual quotation in the article, said he is not in favor of a world of “Privacy above all things.”

He did not say that preventing encryption is above all else. In other words the correct way to parse the paraphrased quotation from the headline is not

‘(prevent encryption)(above all else)’

but

‘(prevent)(encryption above all else)’

The distinction may be lost on the cypherpunks (and I don’t mean that as an insult) who frequent this blog. For the record, I don’t particularly agree with Steinbach or Comey. Nor do all lawmakers:

http://www.washingtonpost.com/blogs/post-politics/wp/2015/06/01/lawmakers-urge-fbi-to-stand-down-on-decryption-push/

And this:

“a bunch of blood thirsty megalomaniacs with zero morals”

We probably don’t want those types in the NSA, the FBI, or the local police, or in a position to enforce international law. But for the most part people in such highly regimented organizations do exactly what their bosses tell them to. And technically the bosses are accountable to the voters, in the free countries, at least. So it’s probably more constructive to call attention to the particular actions of our elected leaders, good or bad, rather than dismiss them all as “a bunch of blood thirsty megalomaniacs with zero morals” and not even bother to register to vote or show up at the polls, or educate ourselves on who is the greater good or lesser evil to vote for.

FBI officials might enforce the law, but they don’t get to make the law.

nacho • June 5, 2015 11:25 PM

@ Justin, Clive

Re:
“Which is why I think cyber should be dealt with by local and international law and appropriate law enforcment, not a bunch of blood thirsty megalomaniacs with zero morals, we politly call “War Mongering Fools”.”

Your kidding me right? The mafia has worse morals. Spanning across borders doesn’t change that.

AlanS • June 6, 2015 9:59 AM

@Mace Moneta
“If they can’t even protect government agencies from the “largest thefts of government data ever seen” by a foreign country, what value is the program?”

Agreed. See U.S. Was Warned of System Open to Cyberattacks for how bad the USG is when it comes to protecting sensitive governemnt data.

In the most egregious case cited by the inspector general, outsiders entering the system were not subjected to “multifactor authentication” — the systems that, for example, require a code that is sent to a cellphone to be entered before giving access to a user. Asked about that in an interview, Donna Seymour, the chief information officer at the Office of Personnel Management, said that installing such gear in the government’s “antiquated environment” was difficult and very time consuming, and that her agency had to perform “triage” to determine how to close the worst vulnerabilities.

Maybe some of that money being wasted on NSA facilities could be spent implementing proper security environments to protect federal data. The government publishes lots of security standards but it appears that federal agencies either can’t afford or can’t be bothered implementing them (although they are required to by law). It seems like the OPM environment would struggle getting authorized under FISMA Low nevermind the security controls that should be in place for that environment.

EmptyWheel on those who live by the sword setting themselves up to die by the sword: Bulk Collection Is All Fun and Games Until Office of Personnel Management Gets Hacked

Once the government does whatever it can to protect the millions compromised by this hack, I hope it will provide an opportunity to do two things: focus on actual cyber-defense, rather than an offensive approach that itself entails and therefore legitimates precisely this kind of bulk collection, and reflect on whether the world we’ve built, in which millions of innocent people get swept up in spying because it’s easy to do so, is really one we want to pursue….China has, unsurprisingly, now adopted our approach, even if it would take a decade for it to catch up in ability to bulk collect from most nodes. And that’s going to suck for a lot of government and private sector employees who will be made targets as a result. But that’s the world and the rules we chose to create.

Mr E • June 6, 2015 10:56 AM

Sounds like a great way to collect zeroday exploits as they first begin to be used, and find compromised machines to (ab)use.

Ross Snider • June 7, 2015 5:19 PM

Schneier – you call for people to think with their heads rather than reacting with their fear.

The NSA absolutely does use these capabilities to protect the United States from cyber attacks – in a cyber war that it is losing if for no other reason than it has more to lose in this assymetric warfare than it can win.

These powers could be used to abuse civil liberties. Are they? Presuming that they are is reacting with fear. There should definitely be work into seeing whether this is the case.

But more broadly the situation we are in exists for two reasons:

1) The types of investments that are required for security in the cyber theater are exactly the same ones that cover the area of civil liberty. There is a contradiction – at an atomic level – between national cyber security and domestic cyber liberty.

Putting an IDS on the internet backbone is BRILLIANT and it’s POWERFUL and if you want your country to be brilliant and powerful we should cheer about this.

It’s not possible to do something that powerful without interacting with civil liberties – given the ‘physics/nature’ of cyber assault.

2) Mission creep, things considered (rightly/wrongly) to be national security concerns arising domestically (Occupy, etc), population control are enticing and real things that incentivize these agencies and its executive and legislative branch to turn its military tools against its people.

We DO want the NSA to run a backbone IDS. We DO NOT want that IDS to interfere with anything we could consider civil liberties.

The US government’s solution (because there is not a technological one, and because they may not care so much for a technological one) is to use policy.

The American people no longer have faith in policy and do not understand even what the word ‘national security’ means (it means strategic interests as well as defense).

But they should not panic. And we should not encourage them to.

Nick P • June 7, 2015 9:46 PM

@ Ross Snider

I’ve easily dispelled this monitoring for security myth before. The starting point is knowing that hackers get in due to bad security. There’s known ways to design systems so secure that remote hackers barely have a chance. The NSA and Defense sector even have such products they could offer at cost (subsized by government) to protect our infrastructure. Yet, they instead push for backdoors, surveillance, monitoring, and secrecy. They also work to prevent Americans from getting ahold of such highly secure systems. If secure computers are only defense, then why would a NSA aiming for national security block Americans’ access to them while wanting more backdoors? Obvious answer: they have a different objective than they say and are selling the means to it as a security solution.

If you doubt it, I give specific examples in this post on how NSA has cleared the path for hackers going back decades.

“We DO want the NSA to run a backbone IDS. We DO NOT want that IDS to interfere with anything we could consider civil liberties.”

NSA stays interfering with civil liberties while possessing criminal immunity. They can’t be trusted. End of story. The only IDS’s preserving civil liberties would be IDS’s mandatory for ISP’s to support law enforcement’s needs when they have a warrant. If the behavior is obviously malicious, the ISP might also offer up the information willingly. Many are doing this with both private security firms and even Microsoft busting huge botnets that NSA did nothing about despite having more visibility. So, I’d say let’s keep trusting the groups not out to snatch our freedoms who have a proven track record on cybercriminals. That’s thousands of times better than NSA’s track record of 1 money-moving plot foiled out of 50+ potentials plus a shitload of lies, abuses, and wasted billions of tax dollars.

Ross Snider • June 8, 2015 12:52 PM

@Nick P

Thank you for taking my comment seriously and responding critically to it.

The starting point is knowing that hackers get in due to bad security. There’s known ways to design systems so secure that remote hackers barely have a chance.

There are complications to this. It is not merely about preventing attacks – it is about detecting them, and also being able to go back in history with newly found thumbprints and find old compromises.

Even with extremely secured networks, it is not so difficult for skilled attackers to work themselves in. In the case of nation states, the incentives/cost-tradeoff is there.

It’s also the case that US industry is not secure in this way. We are talking about the internet – not a closed DoD network.

The NSA and Defense sector even have such products they could offer at cost (subsized by government) to protect our infrastructure.

There is a lot of this that is done. The energy sector, for example, has had a huge amount of help with cyber security. This is also true of the finance sector.

Yet, they instead push for backdoors, surveillance, monitoring, and secrecy. They also work to prevent Americans from getting ahold of such highly secure systems. If secure computers are only defense, then why would a NSA aiming for national security block Americans’ access to them while wanting more backdoors? Obvious answer: they have a different objective than they say and are selling the means to it as a security solution.

They do both things. They legitimately protect the United States and monitor American peoples. The DoD pursues anything it can to get national strategic advantage. This includes limiting technology available to citizens. The calculus performed by the DoD is one done to maximize national security and strategic influence/power. In this case having these sorts of large surveillance systems do increase national power. Yes it hurts civilian power.

If you doubt it, I give specific examples in this post on how NSA has cleared the path for hackers going back decades.

I’m familiar, but interested to see what you will post.

“We DO want the NSA to run a backbone IDS. We DO NOT want that IDS to interfere with anything we could consider civil liberties.” NSA stays interfering with civil liberties while possessing criminal immunity. They can’t be trusted. End of story. The only IDS’s preserving civil liberties would be IDS’s mandatory for ISP’s to support law enforcement’s needs when they have a warrant. If the behavior is obviously malicious, the ISP might also offer up the information willingly. Many are doing this with both private security firms and even Microsoft busting huge botnets that NSA did nothing about despite having more visibility. So, I’d say let’s keep trusting the groups not out to snatch our freedoms who have a proven track record on cybercriminals. That’s thousands of times better than NSA’s track record of 1 money-moving plot foiled out of 50+ potentials plus a shitload of lies, abuses, and wasted billions of tax dollars.

Lots to respond to here. Microsoft and others have been complicit in adding backdoors to services and themselves do not have visibility into federal use of data. Nor is that their role (we don’t want a profit-maximizer to worry about our civil rights, because profit-maximizers will only champion civil rights if it maximizes profit).

The NSA does plenty – do you hear about it? Nope. You can’t assume that they don’t do anything because you don’t hear about it. If you look through the Snowden docs you can see (some) examples of activities that they have performed…

But this is beside the point. I agree that we might not trust the NSA.

Yet it remains true that we do want to run an IDS on the internet backbone. We just want to be able to trust or technically enforce that it is not abused. Maybe you trust profit-maximizers (I don’t). Maybe you trust a court system (I don’t). Maybe you trust the NSA (I know you don’t). But the point is that we want to find some way to do it. The question is whether this is a case of having our cake and eating it to.

I’m not sure that it is. I feel like there’s got to be some arrangement whereby we can get both.

Clive Robinson • June 8, 2015 5:11 PM

@ Ross Snider,

I’m not sure that it is. I feel like there’s got to be some arrangement whereby we can get both.

Sorry, privacy thus security is like virginity, either you have it or you don’t. There is no middle ground, nor if you think about it for a little while can there ever be.

There is a truism that “Three people can keep a secret, only if one of them kills the other two”. This has been attributed to a fairly famous US political figure, who very probably had sufficiently intimate knowledge of both secret keeping and betrayal to make the truism.

Humans have a failing that appeared to start around 50,000 years ago when male testosterone levels started to drop and thus what we sometimes call “their feminine side” came through. Whilst this enabled the expanding of social groups beyond the immediate genetic family it has a flip side.

Thus the “coin that is society” has two sides “trust” and “betrayal” and each time people meet they all “flip the coin”. However the coin is unnecessarily loaded in favour of “trust” which whilst it has enabled society to develop had ment utter devistation and death for those who get betrayed.

Why do people betray others trust, well it all boils down to the “greed” that the greedy call “Power” or “Status”. The interesting part is that power and status or the root of them is “given” not taken. For instance both a king and a begger are human with all the dusual frailties as Shakespeare noted “prick me and do I not bleed?”, thus anyone can do away with another on a mear whim if they wo chose. The power and status they may or may not have is accorded to them by others.

As I’ve pointed out in the past, perhaps the best managers of people are those in charge of charities, who get unpaid volunteers to do mainly drudgery work for nothing, not once, not twice but over and over again day in day out. Importantly they do this by making the person feal not just good for having done the work, but obligated in some way to do it again.

It would appear that most people have buttons that can be pushef and different perceptions of risk and reward. Thus if you can push a persons button and make them feal grateful you have done it then you will be gifted with that persons labour and trust.

In the past fear of strength made people align with that strength and possibly without realising increase the strength thy fear. It is this that gives us the “Authoritarian and authoritarian follower” whilst much research has been done on the former, considerably less so on the latter.

Likewise status has a similar effect, a person who is perceived as having high status, has others of lesser perceived status aligning with them. In the process those of the lesser status actuall give more status to the higher status person, for what is at best only a marginal fraction of the collective status.

We joke about people working with other peoples money in the hope some of it rubs off onto them. Well people get entrusted with handaling money either because they are belived to be trustworthy, or they are belived to be able to provide better returns than other people. And it’s getting the mixture of trust and greed right that con artists use to get rather more than a bit of the money rubbing off in their favour.

Which tells us why there will always be betrayal, as long as one person can find the right button to push they can get another to betray a third person.

The only way to reduce the risk of another betraying you is to have more leverage against them than others can get. This is the idea behind crime gangs, where you become “made” by committing a vary serious crime such as murder for the “boss”. You know that you then have to trust the boss because he can betray you to the authorities to be –in times past– executed. The boss knows that as long as he does not give those beneath them sufficient leverage then they will not betray them to the authorities. However the boss is also aware that there is a price on their head, if they can not deliver the rewards those beneth them want then they will be come prey to the maxim “dead men don’t talk”. The authorities are aware of these rules and thus move the goal posts with “Kings evidence”, to which the counter is the boss going after a betrayers family, so the authorities have “witness protection” not just for the betrayer but for their immediate family.

Thus you can see why the original truism is a variation of “dead men don’t talk”, and in order not to have to deal with the untidiness of having to permantly silence someone it’s far easier not to tell them your secrets in the first place, thus they can not betray you. The downside is that without some degree of trust you can not function in society, society generally won’t let you as it needs trust to function. Thus you have to put strong limits on the trust you do give, alowing a third party to “arbitrarily take away” those limits in any way is a recipe for disaster waiting to happen.

Society requires trust not to be betrayed for society to function, without trust there can be no trade or commerce of any worth, nor privacy. Thus those who give –spurious– argument as to why you must not have privacy are actually “mad men hell bent on destroying society” and should be treated as such. The founding fathers realised that for privacy you must have security in your thoughts, papers, possessions and places, and that, that security should only be rescinded on an individual basis with sufficient cause that others would be harmed if it were not.

There is no “middle ground” allowable, to do so would betray society at a foundational level and it could not survive the loss of it’s major foundation.

Justin • June 8, 2015 7:43 PM

I see some fairly heavy arguments being put forth here, so I think I may as well add my 2¢.

@ Nick P

I’ve easily dispelled this monitoring for security myth before. The starting point is knowing that hackers get in due to bad security. There’s known ways to design systems so secure that remote hackers barely have a chance.

Ok, “hackers get in due to bad security” and you are pushing high assurance as a solution. I just read a bit about those “known ways to design systems so secure that remote hackers barely have a chance,” (to be exact, the seL4 project.) They involve automatically checked formal mathematical proofs of correctness, data integrity, and data confidentiality. I can’t even begin to imagine the highly exacting standards of software engineering, the incredible amount of work, and the highly specialized expertise needed to do that kind of development. As it was, it was apparently ten years too late to market, the company went bankrupt, and they ended up open-sourcing the whole thing (undoubtedly much to the consternation of the NSA.) And that’s just a very basic, bare-bones operating system kernel. None of the “stuff” that would run on that to make up a full-blown operating system. No databases. No web apps. No web browser. No email.

I like the idea of high assurance, but I’m playing devil’s advocate here, because if you are going to push that as a solution, I think you should do better (i.e. get into more details, rather than short sound bites) to dispel the ‘myth’ that it’s just incredibly impractical for the average programmer or user. (I don’t really know much about high assurance, and I would definitely like to learn more about it myself.)

In the meantime, the world has not yet deployed high assurance on the Internet, and we need to be pragmatic about actual threats occurring “in the wild.” So if the NSA can run an IDS on the backbone and do something to alleviate those threats for Average Joe, I say, “Bully for them!”

In some of your posts you seem to have plans to build something high-assurance. Personally I think you need to specialize. Somebody needs to build high-assurance hardware, and somebody else needs to assume the hardware works correctly and build high-assurance software. There should not be a concern (as you have sometimes expressed) of subtly undermining a high-assurance software development process because of low-assurance hardware. That is not a critical path, because once high-assurance hardware is available, the software can be bootstrapped and formally verified on it. For example, (of critical vs. non-critical path,) the developers of seL4 were able to use a low-assurance compiler because they had developed other means of formally verifying the correctness of the compilation. I am interested in your thoughts on these things.

@ Ross Snider

Yet it remains true that we do want to run an IDS on the internet backbone. We just want to be able to trust or technically enforce that it is not abused. Maybe you trust profit-maximizers (I don’t). Maybe you trust a court system (I don’t). Maybe you trust the NSA (I know you don’t). But the point is that we want to find some way to do it. The question is whether this is a case of having our cake and eating it to.

As far as trusting a court system, and how far we should trust it, that is the reason for the constitutional limits that have been enforced to what is admissible for law enforcement to present in court. Can the NSA run an IDS while maintaining the ethical principles of the 4th Amendment? I don’t see why not, but I do insist that those principles upon which our nation is founded be upheld.

@ Clive Robinson

Sorry, privacy thus security is like virginity, either you have it or you don’t. There is no middle ground, nor if you think about it for a little while can there ever be.

That view is very absolutist. In real life we have to share some information with some people, which we prefer they not share with other people, but we only have extremely limited control over what happens with our information once we have been obliged to share it with some entity outside our absolute trust. Bruce just wrote a whole book about it.

Nick P • June 8, 2015 9:27 PM

@ Justin

Focusing on the main critiques.

re my post

My article showed evidence that U.S. intelligence agencies and LEO’s monitoring is not the answer to security. Also that their INFOSEC proposals can’t be trusted at all if aimed at general public, business, or non-Defense government. It supported this by defining security and showing how NSA opposed it at every turn outside Defense. High assurance is orthogonal: only the security critical components need to be strongly assured and many can be standardized with public investment. DARPA, NSF, and others are doing that now with their R&D. Whether high assurance or not, my article still stands given that trusting a subversive or lying organization is totally unacceptable. Even worse if it’s goal is to breach security and yours is to enforce it.

re commercial prodocts

One of the earliest to the market was Burroughs B5000 in 1961. It has two bit tags in memory where it protected both pointers and code from unauthorized manipulation. It also provided isolation and protected interfaces at app level. Every software-based code injection I know of off the top of my head relies on violating one of these principles. Enforcing them with cheap circuits stops about all attacks on implementation outside covert channels. Recent academic work built similar architectures with some having single-digit performance hits and even legacy compatibility.

So, the truth is totally the opposite of what you suggest if you focus on making the machine protect its integrity by design. Even NSA Technical Director Brian Snow said that in a public presentation. Then, the tools convert the source into something that takes advantage of that. The developers just have to follow some good software and security engineering practices. The result is 99+% of attackers can’t do anything past DOS, the rest might be contained/detected, and leaks/configuration errors become the norm. Companies and academics have produced tools to handle that more easily, as well. Yet, even if we just make a few changes, we can eliminate most of our risk and make code injection something only pro’s will try. Automated diversity tools further convert those attacks from “one flaw to rule them all” to “ok, let’s hope we can figure out this configuration before IDS flags us.”

Conclusion

The ability to make life hard for attackers goes back to 1960’s technology. The 1970-1980’s security kernel and capability technology did more while focusing the assurance on the small TCB’s. The modern stuff goes across the board. The U.S. I.C./L.E.O. community can solve most of our problems by working with industry to (a) modify the hardware architectures, (b) compile legacy systems to them where possible, and (c) develop tools to make the rest easier. Instead, they deny us all the security technology they have, push stuff they know is insecure, add weaknesses, and try to vacuum up everything in territory friendly and enemy alike. All in all, NSA and FBI do anything but trying to protect us in the ways they know will work.

We need real INFOSEC instead. Good news is that we’ve come a long way since 1961. I’m sure with a little work we can do at least as well as such systems. We’ll just have to get it from organizations with opposite incentives as NSA and their pals.

Buck • June 8, 2015 10:16 PM

@Ross Snider

we don’t want a profit-maximizer to worry about our civil rights, because profit-maximizers will only champion civil rights if it maximizes profit

While I absolutely agree with that statement, I think you’d be hard pressed to make a claim that the NSA (or any existing IC entity) is not a profit-maximizer… Sure, that probably wasn’t the original intent of such agencies, but we’ve come a long way since then! Much of their (taxpayer funded) work is in fact outsourced to private for-profit corporations, and the leaders of those organizations personally benefit a great deal from the comfortable revolving-door policy. If they’re not actually directing contracts to ensure a lucrative future employment opportunity, they are profiteering from (taxpayer funded) state-secrets.

The corporate takeover of U.S. intelligence (June 1, 2007)

More than five years into the global “war on terror,” spying has become one of the fastest-growing private industries in the United States. The federal government relies more than ever on outsourcing for some of its most sensitive work, though it has kept details about its use of private contractors a closely guarded secret. Intelligence experts, and even the government itself, have warned of a critical lack of oversight for the booming intelligence business.

On May 14, at an industry conference in Colorado sponsored by the Defense Intelligence Agency, the U.S. government revealed for the first time how much of its classified intelligence budget is spent on private contracts: a whopping 70 percent.

Six years later, not much had changed:

Meet the contractors analyzing your private data (June 10, 2013)

With many of these contractors now focused on cyber-security, Hayden has even coined a new term — “Digital Blackwater” — for the industry. “I use that for the concept of the private sector in cyber,” he told a recent conference in Washington, in an odd reference to the notorious mercenary army. “I saw this in government and saw it a lot over the last four years. The private sector has really moved forward in terms of providing security,” he said. Hayden himself has cashed out too: He is now a principal with the Chertoff Group, the intelligence advisory company led by Michael Chertoff, the former secretary of Homeland Security.

and a year after that…

Could Keith Alexander’s Advice Possibly Be Worth $600K a Month? (June 24, 2014)

Think of how much actual security they could buy with that $600K a month. Unless he’s giving them classified information.

And none of them have any real incentive for securing the masses… Well, the Anthem and OPM breaches might change that equation a bit..?

Clive Robinson • June 9, 2015 6:21 AM

@ Justin,

That view is very absolutist. In real life we have to share some nformation with some people, which we prefer they not share with other people, but we only have extremely limited contro over what happens with our information once we have been obliged to share it with some entity outside our absolute trust.

Sadly or thankfully depending on your view our current society and the people within it can not function with either perfect trust or perfect privacy.

However that does not change the fact that in this current society you can not have perfect privacy / secrecy. This gives us two choices, firstly change the society we chose to live in as either individuals or groups, secondly reduce and mitigate the level of trust we give others.

But as I noted society requires both trust and privacy to function, it thus might be true of all types of society we might chose to live in.

Part of the problem is “ownership” that is you corral some part of the worlds resources for your personal use. How do you prove your entitlement to the exclusive use of those resources to second and third parties?

We currently do not know of a way to do this and maintain absolute privacy.

Thus you come back to mitigation of that “required” loss of privacy and the “trust” you have to place in society that amongst other things arbitrates “ownership” through an –assumed– impartial third party.

The problem is that as can been clearly seen in other countries currently “preferential impartiality” can be “bought” if the price is sufficient and the payment method used is sufficiently secret. It’s so common we call it bribary and the more generalised process corruption. Thus we see that betrayal and secrecy tend to go hand in hand which is why society demands a level of trust and a reduction in privacy as a consequence of it’s needed impartiality to adjudicate. You can go through all aspects of our current society and show this trade off between “required trust” and “privacy”.

Thus if you are going to be part of our current society, you can not participate without placing some measure of trust in the instruments of society.

Thus you arive at the conclusion that there has to be trust and some loss of privacy for society to function and at some point there will be betrayal, you might view it as “absolutist” but it is the reality we currently live in.

Thus comes the questions of cost and mitigation we currently see being played out. The FBI want to minimize their cost by making privacy impossible, other more sensible members of society know that without privacy fraud and theft would destroy society. Thus a balance has to be struck between privacy / secrecy and trust, the Founding fathers and to a certain extent the English Barons who forced the King into signing the first of the “Grand Charters” came up with the notion that loss of privacy could only be with good cause and on a case by case basis. What the FBI and other agencies of government want is to return to the times of Absolute Power via “Divine Right”, which history tells us can only lead to civil unrest and destructive conflict.

Nick P • June 9, 2015 8:40 AM

@ Ross Snider

“There are complications to this. It is not merely about preventing attacks – it is about detecting them, and also being able to go back in history with newly found thumbprints and find old compromises.”

This is possible. Yet, I don’t recall a Snowden leak talking about this. About all of their equipment is setup to filter communications based on criteria they know in advance. That precludes what you’re describing. Also, remember the scale of their collection and it explains why.

“Even with extremely secured networks, it is not so difficult for skilled attackers to work themselves in. In the case of nation states, the incentives/cost-tradeoff is there.”

Certain designs have never been compromised during production use. Further, the high assurance products NSA et al use only get certified after at least a year of pen testing by their own people. Further, there’s other designs nobody has shown how to compromise. All have in common that they force the attacker to use a physical attack. That narrows the number of attackers considerably.

Of course, you combine these with local IDS’s for spotting anything out of the ordinary. With the right setup, most attack types are impossible and what is left looks suspiciously like bad guys.

“The energy sector, for example, has had a huge amount of help with cyber security. This is also true of the finance sector.”

The good stuff they use is private sector. The DOD/NSA stuff that’s secure is forbidden to them, too. The stuff that they were given is certified EAL4: the same as Windows and other systems that get hacked routinely. Matter of fact, one of the criteria for that is that you can’t connect it to a network and everyone in the system must be trustworthy. U.S. government giving banks that is setting them up to fail. Fortunatley, breaches have mostly been minor as hackers target business’s less secure bank accounts instead.

“They do both things. They legitimately protect the United States and monitor American peoples.”

Please cite the evidence. Our side pushed them hard until a bunch admitted it was maybe 50+ plots. That got down to 1 plot of moving money to a foreign country. Further, they said they were protecting our INFOSEC while Snowden leaks show them weakening them across the board and good stuff is still illegal to buy. That’s despite our enemies hitting stuff right now. They also forbid this security while preaching we must prepare for massive cyberattack. Either they’re lying about the attack or willing to let our entire country get smashed for convenience of their domestic surveillance.

So, in terms of activities and results, they haven’t delivered shit except more body bags for the enemies to put our IT systems in. 😉

“Microsoft and others have been complicit in adding backdoors to services and themselves do not have visibility into federal use of data. Nor is that their role (we don’t want a profit-maximizer to worry about our civil rights, because profit-maximizers will only champion civil rights if it maximizes profit).”

That’s true and the companies’ credibility should be assessed. Microsoft is obviously as devious as NSA. However, there are other companies (Qwest, Lavabit) that tried to protect users privacy and security who paid a huge price for it. The most credible and caring company in U.S. is simply not allowed to secure their users in a police state demanding backdoors. There’s very few exceptions and none of us even know how they pull it off.

“But the point is that we want to find some way to do it. The question is whether this is a case of having our cake and eating it to.”

It’s a reasonable point you have but it undermines itself when you know a single fact: there ARE IDS’s all across the ISP’s and backbones. It’s occasionally helped them detect or stop very obvious stuff. Examples include botnets with known IP’s, data exfiltration heading from U.S. servers to Chinese IP’s, and so on. Each of these are simple cases that don’t support much utility for general case. Also, they all worked because the IDS was inside ISP’s systems to get a clear view of its network (minus carrier NAT etc).

So, I’m all for IDS’s at the carrier level. Even Tier 1 would be fine if it’s the ISP’s doing it with regulations on the data. Yet, most productive monitoring is at Tier 2 and below. Most of them already do it. They mostly use it to defend themselves. New regulations could be used to, at the least, inform users of activity at their network that looks like a leak or attack. Doing it the way that works is probably better than the massive expense and risk of an unproven Tier 1 solution.

That said, they might make a better case if they did an experimental demonstration with a Tier 1 provider showing their success stories. A NSF, DARPA, or DOD-funded demonstration of the tech that just tried to catch as much as it could. The results would be independently verified by numerous parties. Their arguments for such collection would gain more weight if (a) it did catch a lot of stuff and (b) it was stuff ISP’s themselves couldn’t catch. I have my doubts but that’s at least a reasonable way for people on your side of it to get started on backbone IDS.

BoppingAround • June 9, 2015 10:32 AM

Clive,

The interesting part is that power and status or the root of them is “given” not
taken.
The power and status they may or may not have is accorded to them by others.

That reminds me a book [A] I read once, on several belligerent Mafia crime families.

Near to the end of the book the protagonist, a soldato, kills the Don of one of the rivalling families. He then describes how his ‘family’ celebrated, how there seemed to be an end to the bloodshed until he realised that if a regular guy like him could kill the most powerful man in the city, what good was all his power? He would have probably still been alive, had he not been so powerful. No matter how strong is someone, there will always be someone stronger to take that someone out.

He continues that ‘greediness is bullshit’: when you are poor, you think a few more bucks is all you need. Then you realise it wouldn’t be bad to have a nice car, you get a great job in some high position but you’re thinking about going even higher. And before you know it, you want to be the president of the country and win a war against Germans.

In conclusion, he’s thinking about changing priorities a little, going over of his relationship with the people he loves, with his friends, and how he wouldn’t like to find Jones (I don’t remember the exact surname; it was the surname of another soldato, the protagonist’s friend) pointing a gun at his head. Seemingly, he couldn’t deal with the possible betrayal and loss of trust. Perhaps that was what made him to sell the family out to feds at the end, committing the betrayal himself.

Sort of an illustration to one part of your post, it seems.

As I’ve pointed out in the past, perhaps the best managers of people are those in
charge of charities, who get unpaid volunteers to do mainly drudgery work for nothing,
not once, not twice but over and over again day in day out. Importantly they do this
by making the person feal not just good for having done the work, but obligated in
some way to do it again.

…by giving volunteers something, whether illusionary or not (say, gratitude, perceived benefit of ‘helping society’ or ‘importance’). Or, by feeding on the justice side, that is, ‘if you are fine, wouldn’t it be good to help a bit those who aren’t?’

Giving that a bit of thought, that’s quite scary.

[A] I don’t really remember what it was: a book, on-line or off-line; a game; a film. Let it be a book.

George • June 9, 2015 7:23 PM

@ Clive Robinson, “As I’ve pointed out in the past, perhaps the best managers of people are those in charge of charities, who get unpaid volunteers to do mainly drudgery work for nothing, not once, not twice but over and over again day in day out. Importantly they do this by making the person feal not just good for having done the work, but obligated in some way to do it again.”

That’s why perhaps the best data collection techniques are those employed by social media in the private sector, who get unpaid users to voluntarily give up their personal data. Personal data includes, on top of vital stats, what people think and thought. The best part is to get people to willingly waive their right to privacy by clicking thru well-versed user agreements.

David Hawthorne • June 10, 2015 1:21 AM

This system was already leaked about. An overzealous “anonymous source” on a major, inter-agency hack investigation just had to explain why “they” were so sure it could be attributed to Russia. The journalist had inside access to multiple teams from multiple agencies. One of those team members became the “anonymous” source.

The journalist and the source received a “talking to”, is what I heard. (I was pointed out this article via an old friend who sends such things by snail mail through his secretary. Just to add some flavor to this. True story, too.)

No fun when your editor hammers you down, and you did not “mean to”.

I am surprised everyone here was oblivious to this, but such is how things go. [Of course, I do not buy this for every poster.]

I disagree that this system falls under the category of “widespread, domestic, warrantless wiretapping”. The system is designed to detect nation state level attacks utilizing zero day vulnerability and highly sophisticated malware signatures.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.