Clive Robinson July 2, 2015 11:35 AM

The Intercept notes,

These servers store “full-take data” at the collection sites — meaning that they captured all of the traffic collected — and, as of 2009, stored content for 3 to 5 days and metadata for 30 to 45 days

It’s six years since then or four compleate generations in computing terms, which means the hardware is going to be atleast 16 times more powerfull with 64 times more storage and around 32 times more bandwidth in communications.

What do you think that has done to that 3-5 days? Turned it into “stored for ever” perhaps?

Flaco July 2, 2015 11:42 AM

Check out the HTTP requests.

Aside from pulling in everything on port 80. Check out what they choose to include in the slides. book searches news searches

me July 2, 2015 12:15 PM

Regarding “Part 2”. Me thinks all that talk of mysql and oper is for laying the groundwork of the future “shocking” news that Snowden made a selector database dump (shocking to “them”, that is, not us).

Joe Buck July 2, 2015 12:25 PM

Yes, it seems that the NSA has never been sure what Snowden actually got, which is why, early on, Greenwald was having such fun waiting for official statements and then pulling out another document showing that the official statements were lies. This probably means that Snowden used this administrator access to bypass any checks or any logging, which of course means that anyone else with similar access can do the same, either for official purposes or their own purposes.

Kent Borg July 2, 2015 12:55 PM

It has been claimed that the Chinese and Russians must have gotten their own copies of the Snowden files by now. So, presumably, would the US.

Have US reactions gotten more graceful at some point, suggesting when they got their copy? (If?)


Jeff July 2, 2015 1:04 PM

It seems that the NSA’s public definition of the term “collected” may be at variance with their internal usage.

Public definition: (reference EFF):
Under Department of Defense regulations, information is considered to be “collected” only after it has been “received for use by an employee of a DoD intelligence component,” and “data acquired by electronic means is ‘collected’ only when it has been processed into intelligible form.”<\a>

Internal definition:
see slide 13 of 20 in “Email Address vs. User Activity”
“I have an Email Address and want to see if it’s been collected.”

Publicly, I thought we were supposed to believe that raw data Was uncollected and was not very usable. It seems to me that such raw data can become collected instantaneously, just by a simple search, making all data virtually collected.

Jeff July 2, 2015 1:13 PM

Clarifying my previous comment: “I have an Email Address and want to see if it’s been collected.” implies that the data has been collected BEFORE the analyst has shown any interest in ti.

Bob S. July 2, 2015 1:19 PM

“FISA warrants have authorized “full-take” collection of traffic from at least some U.S. web forums.”

Is this forum on the full take list?

Mass surveillance = mass control = mass domination

If you think this through a bit, Five Eyes has granted itself entitlement to become an electronic dictatorship of the entire world. They have been allowed, or took the power, to become an at least parallel government run by military generals, not elected representatives. It is unimaginable to me they do not realize their power.

They can do it because they can, because they act in secret, for our own good they say and have sabotaged or co-opted key government officials and agencies all over the world.

Noted above the FOREIGN intelligence court granted NSA/Five Eyes power for a “full take” of USA WEB FORUMS.

How could that be??? USA controlled by the foreign court?

Are there any REAL limitations at all?

More later….

d33t July 2, 2015 1:34 PM

“In a statement to The Intercept, the NSA said:

“The National Security Agency’s foreign intelligence operations are 1) authorized by law; 2) subject to multiple layers of stringent internal and external oversight; and 3) conducted in a manner that is designed to protect privacy and civil liberties. As provided for by Presidential Policy Directive 28 (PPD-28), all persons, regardless of their nationality, have legitimate privacy interests in the handling of their personal information. NSA goes to great lengths to narrowly tailor and focus its signals intelligence operations on the collection of communications that are most likely to contain foreign intelligence or counterintelligence information.”

“When systems administrators log into XKEYSCORE servers to configure them, they appear to use a shared account, under the name “oper.”

“Systems administrators, however, are able to run MySQL queries. The documents indicate that administrators have the ability to directly query the MySQL databases, where the collected data is stored, apparently bypassing the audit trail.”

From Wikipedia:

“In terms of access, an NSA press statement reads that there is no “unchecked analyst access to NSA collection data. Access to XKeyscore, as well as all of NSA’s analytic tools, is limited to only those personnel who require access for their assigned tasks.” and that there are “[…]stringent oversight and compliance mechanisms built in at several levels. One feature is the system’s ability to limit what an analyst can do with a tool, based on the source of the collection and each analyst’s defined responsibilities.”

From WaPO:

“In another case, the Foreign Intelligence Surveillance Court, which has authority over some NSA operations, did not learn about a new collection method until it had been in operation for many months. The court ruled it unconstitutional.”

“The Obama administration has provided almost no public information about the NSA’s compliance record. In June, after promising to explain the NSA’s record in “as transparent a way as we possibly can,” Deputy Attorney General James Cole described extensive safeguards and oversight that keep the agency in check. “Every now and then, there may be a mistake,” Cole said in congressional testimony.”

From NSA’s website:

“Executive Order 12333, as amended, requires Intelligence Community elements to report to the IOB, in a manner consistent with Executive Order 13462, as amended, intelligence activities they have reason to believe may be unlawful or contrary to Executive Order or Presidential Directive. These reports are also provided to the Office of the Director of National Intelligence. In general, each NSA report contains similar categories of information, including an overview of recent oversight activities conducted by NSA’s Office of the Inspector General and the Office of the General Counsel; signals intelligence activities affecting certain protected categories; and descriptions of specific incidents which may have been unlawful or contrary to applicable policies. The vast majority of compliance incidents involve unintentional technical or human error. In the very few cases that involve the intentional misuse of a signals intelligence system, a thorough investigation is completed, the results are reported to the IOB and the Department of Justice as required, and appropriate disciplinary or administrative action is taken (a publicly available letter from NSA’s Inspector General to Senator Charles E. Grassley on September 11, 2013, discussed twelve instances of intentional misuse that occurred between January 1, 2003 and September 11, 2013).”

Where is this promised “multiple layers of stringent internal and external oversight”in allowing “oper” access to MySQL queries via SSH? I don’t see this as a design flaw or mistake, but a back door for intentional / unauditable violations. So they in fact have no way of really knowing anything about queries done on US Citizen’s data at all via this intentional backdoor. This goes for content of some telephony (cell to VoIP, PSTN to VoIP etc) as well as the content of any other Internet based communications.

NSA takes marching orders from the President correct? Last I heard, he is a “Constitutional Lecturer”.

Spaceman Spiff July 2, 2015 1:34 PM

If you type it into your computer (even this message), consider it “owned” by the NSA…

30f49if July 2, 2015 1:51 PM

@Spaceman Spiff: One of my favorite offline spying techniques, which doesn’t have a single reference on the internet for some reason, is how MS sends MUI cache data to it’s data centers on update requests so it can map the source of binaries world-wide..

They actually use this to help bust malware authors world-wide and map authors of software.. They most likely correlate it to search and account data on the back-end to profile some more..

I also assume microcode backdoors. Like MMU backdoors governments use for privilege escalation but only on focused attacks not APT malware thus to not expose backdoors in hardware..

Windows also builds offline databases for file system searches and the new apps that get synced without user permissions..

These things don’t make the headlines for some reason..

Pedro July 2, 2015 2:31 PM

From the XKS admin’s handbook: port: 2412 port: 2412

Above resolve to same IP. The NSA appears to have put a honey pot on the IP 🙂

nmap one of the above FQDN for fun.

Edward Snowden July 2, 2015 2:37 PM

If a mysqldump –all-databases > snowden.sql was done on this system…. Whoa.

Bob S. July 2, 2015 2:37 PM

@d33t et al

“NSA takes marching orders from the President correct?”

Apparently not, and that hasn’t been true for a long, long time. That’s one of the BIG problems.

“NSA goes to great lengths to narrowly tailor and focus ….”

It’s not hard to comply with rules you impose on yourself, if you not only make your own rules but the exceptions also and are allowed by law to lie to anyone about the rules and violations and even if caught there are no meaningful sanctions to be imposed.

In short the only operative rule is, the end justifies everything. That’s not the way a democratic society limited by a Constitution is supposed to work.

The rest isn’t hard to figure.

Jacob July 2, 2015 3:55 PM

From the body of the article:
‘Windows Update requests appear to fall under the “update_service/windows” appID’

Why would they need to target Win UD? I don’t assume that this is in order to assess the computer protection status since the collection is limited to only a few days, while the Windows protection status is due to cummulation UD process over many months.

Is this for injecting malware into the UD load thus gaining Admin privilages on the updating machine?
If yes, they must have the MS signing keys in their possession.

Jacob July 2, 2015 4:18 PM

From the “Free File Uploaders” sector document, pages 27 -> end, it appears that the analysts have the facility to add to the search activity a dummy foreign IP/Country Code to be “USSID18 compliant”.

Curious July 2, 2015 4:36 PM

Some thoughts on this verb “collecting” and this noun “collection”:

Regardless of the intent, motive or the lack thereof, for which any information is associated with as having been subjected to monitoring, surveillance, recording, or any collecting/collection in any way, such information is to be regarded as potentially ‘privacy invasive’ regardless of the subject related to such information. This is putting the problem of wanting to respect other people’s privacy before adhering to the needs of others, and before considering anything about the use value of any type of past, present or future information.

This way, the notion of surveillance, monitoring and collection of any type of information, is understood as being ‘surveillance’ as such, regardless of how the information is used or not used, at any point in time, and regardless how such information came about at all. This is putting the problem of “what is surveillance” before determining when anything is or is not surveillance. Presumably, some things would be rather unproblematic (cops incidentally looking at your car or your car’s registration plate), while the internet of things are highly problematic.

This consideration is as broad of an idea and as generalizing as I can come up with, as “information” here is to be understood as ‘anything’ (“any thing”) collected for, assumed for, or otherwise being projected as information for any one or more individuals and/or persons. Ideally, the ‘knowledge’, ‘doctrine’ or ‘theories’ about the creation of ‘information’ as such about any one or more individuals and/or persons, is to be considered irrelevant for learning of the importance of any kind of such information. That way, it should be possible to have a discussion about why information is to be collected, surveilled or monitored in the first place, and for discussing why information was collected, surveilled or monitored in any other case.


‘monitoring’ = someone/something having a focus on detecting and recording the presence or absence of someone/something (some information, a person, people, types of meta-data)

‘surveillance’ = someone/something seeking, or simply gaining information, about the presence or absence of information associated to a person, to people, or to any type of meta-data (categorical stuff, forbids cross checking and connection making).

I haven’t slept on that which I wrote here, but I’d like to think it makes good sense. I would rather be spared of hearing any vague or ambiguous language when it comes to learning about official policies. And it would be imperative that any kind of surveillance (in broadest terms) with regard to communications and technology was basically considered illegal regardless of intent, motivation or lack thereof.

If there was to be an intended exchange of information between two parties, then a reversal of that decision should always be possible as a part of that decision for making that exchange (or information effectively being eternally ephemeral), so that owning or possessing said information once again basically becomes something illegal.

Nothing of what I wrote here relates to any problems of merely being in possession of information, but passing information around would in this context basically be something illegal (too bad Google et al). With all of this being technology and surveillance centric, kids writing down car registration plates on a piece of paper won’t be guilty of any crime.

gordo July 2, 2015 7:33 PM

@ Curious,

Though both are true, I go with the understanding that data is property rather than privacy is a right. Arguing over privacy and when it does or doesn’t happen are “in the eye of the beholder”; “I know it when I see it”; “one person’s privacy is another person’s panopticon” kinds of arguments. They just keep going ’round and ’round in infinite loops of subjectivity. The ‘data is property’ tack seems more concrete, and objective.

While you may not agree with that, you might, however, find the article below, about the ACLU v Clapper ruling, more persuasive. It certainly helped me not lose any sleep over how someone else might define their privacy 😉

In Holding NSA Spying Illegal, the Second Circuit Treats Data as Property
Jim Harper | CATO at Liberty | May 7, 2015

Two points from different parts of the opinion can help structure our thinking about constitutional protection for communications data and other digital information. Data is property, which can be unconstitutionally seized.

The links in the article to other materials are worth taking a look at, too!

You lost the spelling bee July 2, 2015 9:54 PM


cummulation -> cummulative

If you’re trying to impress us with your vocabulary, it isn’t working.

Curious July 3, 2015 3:00 AM

If you say “data is property rather than privacy is a right” then your interest in other peoples right to ‘privacy’ isn’t interesting to me I have to say.

Btw, I vaguely recall there was a poet that made fun of subjectivity as if being something concrete like property. Iirc things aquired by gaze when looking at things behind windows would swirl around when multiple onlookers admired them. Thus, subjectivity is never subjects to being something pragmatic, and neither is the real need for privacy, and so privacy is as real, as people are real (just not property). Thinking of data as being mere “property” is imo questionable at best, and such a regard is perhaps confused with constitutional rights, as if the right to property as such was fundamental (pretty sure it can be argued that you don’t have a constitutional right to something as specific a owning the moon as property for example). I am guessing here, but presumably the US courts will or have tried being pragmatic about it, by effectively dissolving privacy issues and recasting that as being a question about property rights. Is there a privacy law in US at all?

I think it is best to consider data as information foremost, so that you can’t just pretend it is some kind of property that nobody has a say over, as it it all boiled down to being pragmatic about it. One could say that data is property but why would you? Is it because you happen to own or crave it? That would be quite the stretch.

I am not a US citizen, so I couldn’t care less about the considerations made by some US court of law. I can’t be bothered to read that verdict, but if the “seized” data there in that quotation meant that the data was aquired by some kind of surveillance, monitoring or processing of meta-data, then obviously I would have to be against simply treating data as property, because I already argued initially that with regard to surveillance, monitoring or processing of meta-data, the respect of other peoples need and right to ‘privacy’ must foremost be understood by itself as a problem before the needs of others, and obviously before the needs of the court.

Hm, now that I think about it, I believe that privacy issues are best considered as being global issues (like other global problems, like warfare), because if any one government won’t give up their aspriation for managing the populace to their own benefit, they they obviously isn’t qualified for dealing with such issues, especially not if being driven or influenced by corporate interests.

Curious July 3, 2015 3:41 AM

I might be at risk at simplifying things too much, but it struck me now that the consideration of “treating data as being information” foremost and not as property, might have some desirable consequences if taken seriously. As an idea at least, I think this particular idea is important.

‘Information’ previously deemed “secret”, when found out in public, can’t be deemed to be secrets anymore (one particular problem), and more importantly, having that knowledge in public as such shouldn’t be considered something criminal (some other particular problem).

Anyone that find their secrets to be out in the open, would probably do their best to avoid any damages to them (which may or may not be sensible and agreeable), but at least this way, it would be clear that “knowledge as information” isn’t to be subject to considerations of such being property or not, regardless of format (presumably data).

Btw, in my country there are laws against making registries of certain kinds of “data”. Not really sure how that stuff was intended to be policed. Anyway, that initiative seem to me to be about about caring for people’s privacy, one way or another. It should be said that the current right wing of politics in my country, is perhaps the reason why there has been a campaign against anyone learning about your income each year, against tradition, purportedly because of the risk of being literally robbed in your house, which to me seem more like a paranoid and unconvincing argument.

  • My spell checker having been set to “UK”, is driving me nuts here. 😐

Dirk Praet July 3, 2015 5:50 AM

Snowden took a lot of flak for claiming that from his desk he could pretty much wiretap anyone if he had a personal email. Many called it outrageous, even insane. If nothing else, the XKS documents prove that indeed he could, and in a capacity of sysadmin bypassing audit by directly querying the SQL database.

it's gone July 3, 2015 7:44 AM

That realisation… permanent ID, permanently linked to everything vaguely recognisable as uniquely yours or even just “a character trait”. That’s damn scary.

C++ microplugins“… Could be used to add all sorts of Filters, Filters that could include something like phrase recognitions to ID and apportion even the most origin-anonymous clear text. That combined with a permanent ID, could possibly mean that your ID will have lots of suspected to be yours text, apportioned to your profile.

Not only that, if they are going through the trouble of composing/computing profiles that create random profile entries by default, why would they not store those profiles? (not as a copy of all profile data, as a sort of “link page”) Surely their capacity to search the connected materials implies they for example, have profiles on every single email address they have.

They have “a file” on everyone?

it's gone July 3, 2015 8:20 AM

They must have a file on “everybody”. The 30days or whatever number of days traffic storage, must be there to run the tools that connect the data and compile profiles.

Thought experiment. Who is

Are they blind because no one used that address in their “storage time window” and so have to task a tool to run and look out for “” from now on. Then once found, task tools to run and find the IP address activity from now on. Then task tools recursively after every find.

I doubt it. It would take a long time to find out who was, and it all relies on being active from within the initial “storage time window” onwards at least one.

Who is
“Don’t know, he hasn’t used that address in 30days, we’ll never know.” Doesn’t seem legit. They would want to know instantly and the overall stated functionality of xkeyscore implies instant access via search. So they must have “a file” on everyone.

Bolded cause, correct my conclusion if my logic is fail.

Peter July 3, 2015 8:36 AM

I don’t think I’ve seen anything written about s/mime encrypted emails. Only PGP encrypted emails.
Or do they handle all encrypted emails the same way?

If not I’m interested in if I only have missed the info or if they don’t see s/mime mails troublesome.

Skeptical July 3, 2015 9:55 AM

Much is made of the surprise of some individuals whose business models rely on being able to track and monetize what every one of their users do. This actually illustrates one of the differences between the NSA and commercial internet companies like Google. The NSA isn’t interested in compiling a profile on every person’s likes, dislikes, whether and when they’re interested in purchasing a car, etc. Instead the NSA seems interested simply in having a collection available that they can use to search out items of relevance to very particular missions.

Frankly the surprise of those individuals is one of the most unintentionally telling parts of the article. The authors included it no doubt to show how much less sophisticated and competent the NSA is than we think – in part Greenwald indulging in the usual adversarial fantasy, and in part highlighting a genuine policy issue.

But what their surprise actually shows is not only the difference between the NSA’s mission and tasking on the one hand, and those of the truly omnivorous internet companies on the other hand, but also the degree to which the NSA’s mission has been so consistently and so deeply misunderstood or discounted by many in possession of these documents.

Once one is able to take seriously the notion that the NSA is extremely driven by its actual missions – and demands made upon it by “customers” – there is little in the article that is surprising.

For example, it seems likely that the NSA is the subject of urgent demands for intelligence on a daily, if not hourly, basis. No doubt some segments of the organization – I hope – are given the space and time to research and plan for the future, but much of the organization must drive to accomplish the mission today, now. A perfect plan three days too late is worthless. And that means scrambling to get together the tools needed into a system that will work “well enough”, notwithstanding the imperfections, notwithstanding a lack of time to consider all alternatives and select the best one.

Consider for a moment the amount of time it took Gellman and his team to go through and understand surveillance conducted on two individuals – a foreign fighter in Afghanistan and his wife in Australia. Imagine how much more time it took analysts to compile that information, to write reports on it, etc.

Now let’s take it up a notch. The target is a key individual in a death squad that divides their time between killing Iraqi Sunnis and planting IEDs to kill coalition forces. Incorporate all intelligence, including latest reports from the field (UAV imagery, HUMINT, etc.), to query most recent data and produce a useable report on any actionable intelligence discovered or on anything that adds to an understanding of the target and his associates. And by the way, you may have to do this while sharing quarters with the very people going out every night on missions, some of whom won’t come back.

Think you have time to aimlessly rummage through irrelevant mailboxes and text messages? I’d imagine not. Think you’d want to do something that triggers the need for an extra report explaining why you ran a selector and describing what now needs to be minimized or discarded? I’d imagine not.

Let’s make it a little harder. You now need, while responding to immediate demands for intelligence, to also massively ramp up your capabilities to accomplish multiple missions like the above and more, and you needed to do so yesterday. Moreover personnel with varying levels and types of expertise need to be able to maintain and use whatever systems you build across the world. And you need to do this while still retaining focus on much more sophisticated and targeted efforts against long-standing targets and adversaries.

I’d expect to find some things that appear to have been selected quickly for immediate utility and then improved and reworked even while operations proceeded. I’d expect to find, for some types of missions, an emphasis on simply having data available for the analyst to immediately begin to work. I’d expect a range of tools.

Remember that the intelligence product ultimately needs to be packaged and understood by a human being who can connect the urgent questions being asked with the information available to him. The US has been deeply engaged in two counterinsurgencies, and in various counterterrorism efforts across the globe, all while coping with rising intelligence requirements related to long-term strategic contingencies and more immediate threats. Even if you thought the absolute worst of the NSA, they don’t have the time or manpower to play Stasi. The kinds of things that some of the comments in here worry over – whether the US Government is sending in puppets to shape the discussion in these threads – are the kinds of things that the organizations in question really don’t have the time or inclination to care about.

It’s a natural human reaction to relate new information we learn, especially about another human being or a human organization, to what we know best, to what we are most familiar with, to what we ourselves care about most. But it can lead to a gross misunderstanding of that new information, to conclusions drawn from it that are false, to an inappropriate focus on risks that are much smaller than we think, and to an inappropriate ignorance of actual risks that we fail to see or consider seriously.

It can also lead to articles that seem to be authored by those who are truly tone-deaf to the material they’re reading. They can see some of the notes and scores, but can’t quite grasp the context which gives them meaning.

gordo July 3, 2015 11:08 AM

@ Curious,

You wrote:

If you say “data is property rather than privacy is a right” then your interest in other peoples right to ‘privacy’ isn’t interesting to me I have to say.

Considering that I actually wrote:

Though both are true, I go with the understanding that data is property rather than privacy is a right.

As but one approach toward accomplishing a common outcome.

My apologies.

Jacob July 3, 2015 1:50 PM

@ Skeptical

The published slides are 5-6+ years old, and I assume they were prepared at the time for internal meetings to show the current state of the art or best practices in intelligence gathering.
It goes without say that since then the NSA capabilities have advanced quite a bit, and a lot of automation and analytics have been added to the program.
The enormous data storage facilities being built, the reported joint programs with domestic LEA, the expansion of the NSA mission to also support Administration’s economic policies, to spy on friendly head of states – all these activities indicate that the NSA manages its time very well and has plenty left for other nefarious activities.

Bob S. July 3, 2015 3:28 PM

In the first Intercept article there are map lines drawn from Fort Meade to what appears to be San Antonio where NSA has a known presence as well as Seattle/Redmond. MS has a large presence in both places, for example MS AZURE relocated to San Antonio.

NSA claims a MS was one of it’s first conquests for PRISM and various sources claim NSA had a backdoor in Windows etc. since 1999.

These days my MS system wants to connect to MS AZURE registered to MS in San Antonio as well as MS Informatica in Brazil and England, constantly.

I wonder what it all means? The amount of connections by MS to everyone’s computers these days cannot all be for updates to AV libraries or patching, in my opinion.

The part that bothers me most however, is there is absolutely no reliable way to get a truthful answer as to how deep NSA is in the MS pocket. Assuming the worst, it would also mean Five Eyes and who knows what other government agencies use MS as a tool for world wide mass surveillance.

tyr July 3, 2015 5:12 PM

I found it curious that they collect windows crash dumps.

If you do any devlopement you crash your machine with
boring regularity so who gets to analyze that level of

I guess if you have a huge budget you don’t care about
wasting time and talent which would accomplish more
and better elsewhere.

Or did Billy Gates talk the government into fixing his
broken crap in exchange for their souls.

BREEZYCRANE July 4, 2015 11:30 AM

  1. vim, rsync, apache and mysql?! So much for the NSA’s unparalleled talent in infosec — xkeyscore is essentially an off-the-shelf redhat system with a point-and-click gui!

  2. ssh? (the very same ssh that they themselves have sabotaged?!) If that’s true, the NSA must have soiled their pants in the days after the discovery of the Heartbleed bug.

  3. shared login credentials?! Are they complete morons or is it an easy way of covering each other’s backs during an audit? I guess their term for it is “plausible deniability.”

  4. these slides are pre-bluffdale. although it is unlikely that the underlying structure of xkeyscore has seen any fundamental redesign, the data processing and storage capacity mentioned in these slides has probably increased quite a bit.

halsey July 4, 2015 11:43 AM

Just a thought: the NSA infiltrates software distributors and standards bodies to undermine cryptography and introduce malware in everybody’s systems, so now that we know the software that x-keyscore relies on, why don’t we return the favor by getting friendly developers at Redhat, SSH, rsync, vim, Apache, etc. to surreptitiously disrupt and/or undermine X-keyscore’s dragnet surveillance?

anonymous July 4, 2015 4:34 PM

What do you think, could Snowden, Snowman, Bigfoot setup kind of backdoor in NSA’s servers?

He able to track his own tracking, methinks… It is useful.

dylan July 5, 2015 3:24 PM


“Or did Billy Gates talk the government into fixing his
broken crap in exchange for their souls.”

Yeah, that one.

Skeptical July 5, 2015 3:51 PM

@Jacob: It goes without say that since then the NSA capabilities have advanced quite a bit, and a lot of automation and analytics have been added to the program. The enormous data storage facilities being built, the reported joint programs with domestic LEA, the expansion of the NSA mission to also support Administration’s economic policies, to spy on friendly head of states – all these activities indicate that the NSA manages its time very well and has plenty left for other nefarious activities.

I’m sure that their capabilities have advanced considerably in 5 or 6 years, as have the various fields relevant to those capabilities.

However, I don’t think that changes the final analysis, especially as adversaries and targets adapt and grow more sophisticated, or as new adversaries and targets develop. Obviously I can’t be certain, and that conclusion would depend on a lot of details, but that’s my overall impression.

We should remember that much of the research being done into the use of new methods and analysis will be done under the uncertain and unforgiving deadlines of war. And while it’s a war in which most of Western society has, fortunately, little contact, it is a war in which much of the NSA is deeply engaged – and it’s a war that must be fought while NSA’s more traditional tasking and missions are worked and accomplished against increasingly capable targets. It’s not a theoretical or merely rhetorical war for those engaged in it.

So I don’t think that advances in capabilities would enable any human beings to slow their tempo – quite the contrary. And I’d expect to see various types of compromises between short and medium term goals and long term goals. I’d expect to see existing tools leveraged to the maximum extent possible, especially during some of the time-period in question. For instance, it may be very nice that someone has a great idea about using a different type of database that is better suited for data that may be in many instances sparsely populated in an analytic space of an increasing number of dimensions, or about how machine learning can be better leveraged – and those ideas may be developed and eventually instituted. But some operational demands probably won’t be able to wait while those things are developed and then deployed.

I’d add that, to the extent automated analytics enables more irrelevant information to be screened before reaching an analyst, this would reduce the degree to which an analyst would intrude into the privacy of the ordinary citizen. Indeed, one of the projects pushed by Binney seems to have had, in my memory of his public description at any rate, precisely this strength.

Finally, as to surveillance of targets like Petrobras, or ministers of finance, these are absolutely legitimate targets for foreign intelligence gathering. One would have to have been completely ignorant of domestic affairs in Brazil for the last few years to suppose that knowledge of Petrobras were not especially relevant to a political analysis of Brazil’s government. And one would have to be remarkably ignorant of economics and finance to think that insight into the position of the German or French governments vis-a-vis Greece, especially in 2011, were not of vital concern to the United States, the UK, and other nations around the world.

These are traditional foreign intelligence targets. There is nothing the least bit controversial among anyone who knows anything about foreign policy in such targeting. The press releases that Wikileaks issues with some of these leaks seem to have been written in some alternate universe, by someone with an extremely weak grasp of the eurozone crisis in 2011.

Nothing brought home to me the fundamental ignorance of some of those at Wikileaks, and elsewhere, who are reading and reporting this leaked material as did a press release a friend showed me yesterday describing a surveillance report on the German preferences for a BRIC infused, all-IMF bailout for Greece in 2011. The release, as I recall, declared that the US would have been “horrified” at the geopolitical implications of this, and asked “what might have been” had the US not had this surveillance report.

What the writer of that press release appears not to realize is that the idea of BRIC nations contributing more to the IMF, and enabling an IMF bailout of Greece that did not require modifications to the operations of the ECB, was old news by the time of that surveillance report (October 2011). The suggestion had been widely reported, as had various other permutations involving funding from China, Russia, and elsewhere. Not only was it old news, it was widely viewed as not feasible given that the actual threat to eurozone in 2011, which was not the possible exit of Greece per se but rather that of Greece’s default undermining confidence in Italy and Spain, rendering it impossible for those nations to continue to finance themselves – THAT would have been disastrous for the eurozone, and indeed the world.

The only real guard against that possibility was to expand the operations of the ECB, or via a very well-funded (beyond the capabilities of the BRIC countries) EFSF, something (in the case of the ECB) opposed by the Germans, and something viewed by France, the UK, the US, and frankly the rest of the world (excepting a handful of other northern European countries) as absolutely necessary.

At this point the more obvious political purpose of some of the reporting of these leaks leaves me with more of a sense of disgust than outrage. No kidding around – it’s saddening to see leaks of such highly, properly, classified material used so – forgive me – stupidly by people who seem dismally ill-informed about international politics. Those people may mean well, and may have the courage of their convictions, but noble intentions don’t magically convey knowledge nor magically endow one’s acts with the desired effects.

Alfred July 6, 2015 11:12 AM

“Or did Billy Gates talk the government into fixing his
broken crap in exchange for their souls.”

That would explain all the “functionality” in Windows 10

gordo July 6, 2015 11:14 AM

Somewhat on-topic:

This video is from 2010. It’s not too hard to imagine that X-keyscore analysts use this kind of tools. I imagine that things have only gotten better since. The three V’s, etc.

Palantir Labs – Graph [01:48]

Palantir in US$20 bn valuation for data analysis
SC Staff | SC Magazine UK | June 24, 2015

[I]nitially the firm built up by serving the CIA, NSA, FBI and at least nine other government agencies.

Report: CIA backed Big Data analytics firm Palantir, raising $500m Series I on $20b valuation
Duncan Riley | Silicon Angle | Jun 24, 2015

Palantir is said to use artificial intelligence to help users quickly analyze different types of data from a variety of sources. It originally set out to detect online organized fraud in the financial industry, but then became a huge hit in the intelligence community, where according to SiliconANGLE’s Maria Deutscher, it would in time earn the designation of Killer App for its role in key operations, and by key operations it’s implied that means serious level spying.

gordo July 7, 2015 8:40 AM

The Intercept’sSOURCES XKEYSCORE DATA” graphic is probably a high-level, tl;dr version of the NSA’s “Driver 1: Worldwide SIGINT/Defense Cryptologic Platform” document.

See also:

NSA infected 50,000 computer networks with malicious software
Floor Boon, Steven Derix and Huib Modderkolk | NRC | 23 november 2013

NSA’s global interception network
P/K | electrospaces | December 3, 2013 (Updated: July 17, 2014)

Leave a comment


Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via

Sidebar photo of Bruce Schneier by Joe MacInnis.