AT&T Assisting NSA Surveillance

Interesting details emerging from EFF’s lawsuit:

According to a statement released by Klein’s attorney, an NSA agent showed up at the San Francisco switching center in 2002 to interview a management-level technician for a special job. In January 2003, Klein observed a new room being built adjacent to the room housing AT&T’s #4ESS switching equipment, which is responsible for routing long distance and international calls.

“I learned that the person whom the NSA interviewed for the secret job was the person working to install equipment in this room,” Klein wrote. “The regular technician work force was not allowed in the room.”

Klein’s job eventually included connecting internet circuits to a splitting cabinet that led to the secret room. During the course of that work, he learned from a co-worker that similar cabinets were being installed in other cities, including Seattle, San Jose, Los Angeles and San Diego.

“While doing my job, I learned that fiber optic cables from the secret room were tapping into the Worldnet (AT&T’s internet service) circuits by splitting off a portion of the light signal,” Klein wrote.

The split circuits included traffic from peering links connecting to other internet backbone providers, meaning that AT&T was also diverting traffic routed from its network to or from other domestic and international providers, according to Klein’s statement.

The secret room also included data-mining equipment called a Narus STA 6400, “known to be used particularly by government intelligence agencies because of its ability to sift through large amounts of data looking for preprogrammed targets,” according to Klein’s statement.

Narus, whose website touts AT&T as a client, sells software to help internet service providers and telecoms monitor and manage their networks, look for intrusions, and wiretap phone calls as mandated by federal law.

More about what the Narus box can do.

EDITED TO ADD (4/14): More about Narus.

Tags: , , , , , ,

Posted on April 14, 2006 at 7:58 AM • 50 Comments

Comments

Thomas Claburn • April 14, 2006 9:31 AM

I hope we find out whether any arrests have resulted from this wholesale data grab. But somehow I suspect the system’s lack of utility is classified.

AG • April 14, 2006 9:37 AM

Holy-Cow that machine is a monster!

Not trying to sound dramatic BUT this machine was built for the sole purpose of wholesale Internet and IP Phone surveillance.

@Bruce
Would it be correct to say that this machine was built to watch only American built IP networks? Don’t networks in other countries run slightly different setups from the group up? Different protocals and such?

Dan Linder • April 14, 2006 9:51 AM

@AG

IP is IP in any country. The only differences the device would have to contend with would be the physical medium. For example, in the US, a “T1” (1.544Mbit/sec) is a common network connection, but in Europe the “E1” (2.048Mbit/sec) is common. The speeds are different, but the primary physical and signaling differences are all handled by different cards in telecom equipment.

So, from the point of view of the Narus device, it just has to talk to a slightly different card to pull the IP “bits” off the wire.

You’re right on two fronts though:
1: It is a monster! 🙂
and
2: It does appear to have the sole purpose of IP surveillance on a massive scale.

Dan

Omar Ahmed • April 14, 2006 10:10 AM

and now, would the gov. oppose large scale cryptography deployment? IPv6? I have no doubt that they would be able to break the codes, but it’s a question of increasing their processing power, and rather being selective in what they tap. Listening to everything wouldn’t be a feasible solution.

AG • April 14, 2006 10:24 AM

@Dan Linder
This device appears to have been created to connect directly to the AT&T backbone network.
The individual backbone carriers must have safeguards in place to keep unauthorized equipment from jumping on their network. Safeguards which I bet are different carrier to carrier let alone country to country.

This machine is more than a directly connected repeator. I am betting it has been created solely for spying on AT&T transmissions.

Most disturbing though is the blatent abuse. The NSA could have got AT&T to install the equipment under the permise that “AT&T” would be watching their own network. This would not break any law since AT&T is allowed to make sure their equipment is not being used for illegal means. AT&T could then have legally passed the information to the NSA.
Instead the NSA went the “illegal” way… of course it really is not “illegal” when President Bush ordered them to do it.

Such a mess.

Pat Cahalan • April 14, 2006 10:42 AM

The Narus-6400 is impressive enough at 622 Mbps, but this NarusInsight box is outside comprehension by mortals.

Something that can monitor an OC-192 at layer 4 in real time.

One machine.

That is an astonishing box (I wonder what the ticket price is on one of these suckers).

To put this into perspective, most of the major telecommunications companies (Qwest, Vario, Adelphia) that get ATM service into the internet backbone don’t use more than an OC-12.

That means that this one box has the capability to fully monitor all of the layer four traffic of 16 major telecommunication companies at once.

If you’re just looking at DS3s, you can monitor 64 of them at once.

I don’t want to be on the internet anymore.

JakeS • April 14, 2006 10:44 AM

Maybe I’m missing something, but is this revelation new or surprising?  NSA has been hoovering national and international communications for decades.  They used to do it for the Cold War, now it’s the War on Terror.  This story just sounds as if they’ve upgraded their kit to keep pace with the growth of the Internet.  Mass internal surveillance may have been illegal in the past but now, as AG says, anything is legal with a presidential order.  And frankly, if you or I were running he NSA we’d surely reckon that scanning everything is something we have to do.

@Omar:  “Listening to everything wouldn’t be a feasible solution.”  Hmm.  With the budget that NSA has to throw at the problem, I’d bet money that they can listen to everything.

Anonymous • April 14, 2006 10:52 AM

TOR is nice and all, but a chunk of the usefulness is predicated on the idea that the entire TOR network can’t be monitored all at once.

This sort of blows that idea out of the water. Sure, the packets are encrypted, but if all of the TOR hosts have their traffic analyzed by a box like this, Big Brother can still figure out where your communications are originating and what the endpoints are.

They might not know what you’re saying, but they know who you’re talking to.

zorax • April 14, 2006 11:01 AM

Do a quick google on Ori Cohen and Narus. It appears that he’s Israeli, and an Israeli venture capital firm is a major investor in Narus. I guess the Dubai port scandal will be forgotten now…

Adam • April 14, 2006 11:02 AM

@Anonymous
Not all TOR nodes are going to be within the reach (technical and legal) of these machines. Add to that that if a node is mediating communications between a whole load of other nodes, it might not be easy to pair them all up.

Pat Cahalan • April 14, 2006 11:02 AM

@ Kevin

Oh, I get that. A large chunk of any telco’s traffic will probably stay “in network”.

I was just using an example to try and bring the staggering numbers into perspective.

If the NSA was sticking Narus-6400s in AT&T’s SF trunk facility, you can surmise that they’re sticking them at every major telecommunications nexus.

Peter Pearson • April 14, 2006 11:02 AM

I’m with JakeS: this is merely an update on an old story, namely CARNIVORE (see Wikipedia). For many years, the US federal government has been forcing phone-switch manufacturers to build in listening ports. Haven’t we all been assuming that the NSA or FBI was reading all our traffic?

Pat Cahalan • April 14, 2006 11:10 AM

@ Peter

Haven’t we all been assuming that the NSA or FBI was reading all our traffic?

I imagine that the general public has been assuming that in fact the NSA and the FBI have not been reading all of their traffic. “Why would they bother to watch me, I’m just Joe Average Citizen? They’re going to be watching those guys who are talking to those extremists over there in Iraq or something.”

It’s one thing for the public to know that the NSA is keeping tabs on who might be talking to terrorists. Most citizens think that this is generally a good idea. However, I think very few of them realize the extent of the surveillance.

Andrew2 • April 14, 2006 11:23 AM

I’ve always thought the volume of data would be sufficent to preclude any direct government spying on me. Frankly, I’m too boring to pay much attention too. But with a box like this, attention becomes quite cheap.

Josh • April 14, 2006 11:59 AM

Many people on this list and elsewhere have remarked that they are “not surprised” that the NSA has the capability to do this, and that they try to do it.

I’m not suprised either, but the big story is
what this means to the relationship between the US gov’t and us citizens.

The president ordered this without legislative or judicial oversite, using as a pretext an “emergency” (*), terrorism, that will probably exist forever.

Power without accountability, forever.

(*) I put emergency in quotes because the word y implies a problem that arises suddenly and requires people to improvise a response. If there’s time to plan, it’s not an emergency.

Bruce Schneier • April 14, 2006 12:39 PM

“Maybe I’m missing something, but is this revelation new or surprising? NSA has been hoovering national and international communications for decades.”

It both is an isn’t a revelation. We know that the NSA is doing this sort of thing, but these tecnical details are new. AT&T’s assistance is also new.

Paul S • April 14, 2006 12:51 PM

Speaking as Joe Average Citizen, if I were to suddenly communicate that I had cheated on my taxes in 2003, and the IRS got wind of it from the NSA, and I HADN’T cheated on my taxes, but just wanted to see if my communications were being monitored, could I be charged with lying to a fed (the Martha Stewart offense)?

Gary Farber • April 14, 2006 1:56 PM

Thanks for the link, Bruce; I’ve added a link to my post to back here, and a link to the dKos post, which I’d not seen before.

Best to Karen.

Christopher • April 14, 2006 1:58 PM

“…AT&T’s assistance is also new.”
Does AT&T have a choice in the matter? What are the consequences of refusing or even challenging such a request?

jon • April 14, 2006 2:53 PM

There are two common ways to duplicate traffic. One is called “port mirroring” and it means that the router itself sends every outgoing packet to two physical ports, one to be transmitted to the real destination and the other to a local system for off-line examination. The second is to use a single port with an external splitter that turns the outgoing stream into two outside the router.

The difference is that port mirroring shows up in the configuration of the router, while an external splitter does not.

So not only was traffic being recorded, but a method was chosen that effectively concealed what was going on, even from other ISP employees.

Nick Lancaster • April 14, 2006 3:23 PM

Apart from the ethical questions of digital McCarthyism …

We’re propagating another illusion. If the NSA is literally tapping the internet and overseas lines through AT&T’s compliance, it follows that we can’t possibly miss a terrorist call.

  • Unless they use code words.
  • Unless they’re not on our watch lists.
  • Unless they don’t use phones or computers.
  • Unless they communicate in person.
  • Unless they don’t need to communicate.

We still need to develop a response that works whether we overhear the right people or not.

Otherwise we might as well be playing mumblety-peg with a chainsaw.

jon • April 14, 2006 4:55 PM

“If the NSA is literally tapping the internet and overseas lines through AT&T’s compliance, it follows that we can’t possibly miss a terrorist call.”

If we believed that, we would be believing in an illusion. However, there are techniques that don’t require us to recognize terrorist calls “as they happen”. One is traffic analysis, that allows us to recognize chatter and then look more closely at the text of the chatter. Another is working backwards from a known terrorist who comes into our hands, and finding out what s/he has been saying and to whom. It would be an error on the part of a terrorist cell to make up a “code book” that we could analyse in the future.

As for the “Unless they don’t use phones or computers.” issue, that is a good point. It’s been standard practice for the British, at least, to dredge up all alien undersea cables at the beginning of a major war in order to force signals traffic into the ether or onto their own cables. I imagine we do similar things today, snooping on non-US cables, scooping satellite traffic, spying on diplomatic bags, maybe even inserting friendlies into human messenger chains.

It’s an arms race, and the fact that you can’t get it perfect does not mean that it’s hopeless.

Nick Lancaster • April 14, 2006 8:32 PM

@ Jon:

I refer you to Malcolm Gladwell’s BLINK, where he details the Millennium Challenge war games, and Lt. Gen Paul vanRiper’s use of unconventional communications – relay runners, etc. … to which the administration said, “You can’t do that!” and reversed the Red Team’s achievements.

The administration then took their ‘win’ and used it as the foundation of their battle plan for Iraq.

It’s that kind of wishful thinking that will cancel out any gains we might reap from this kind of signals intelligence.

109 • April 15, 2006 11:46 PM

  1. Hardware
    I wonder why people are so impressed with capability of monitoring 10 gigabit per second. Google searches through 10 billion web pages in a second. Yes, 10 gigabit per second is impressive in one box, but guess what! There is no need to do it in one box besides impressing NSA officials. The task is inherently parallelizable. A few hundred regular 1K/box PCs would do the job just as good. The difference is that NarusInsight probably costs 10 to 100 times more than a few hundred regular PCs.
  2. Software
    The best software companies (Google, Microsoft) are working on the problem of semantic analysis of the web. And we’ll be there eventually, but we are not there just yet. Try to ask Google something that should be astonishingly simple for the AI to analyze, like http://google.com/search?q=how+many+people+live+in+seattle

Did you get the straight answer? Not really. Now, being an insider in software industry, I know there is no way that Narus has a software that can do anything better than a keyword search. Correlated keyword search, whatever. But no semantics.

Tom Chiverton • April 16, 2006 8:02 AM

Like the man(http://www.imdb.com/title/tt0114558/maindetails) said "The issue’s not whether you’re paranoid … I mean look at this shit, the issue is whether you’re paranoid enough".

Reads like a frigging movie plot – top secret presidential orders, super-high tech computers gathering bonkers amounts of data on silly numbers of people, and doing mind boggling amounts of digging on that.

And all this dubious under US law.

Snakes on a plane, people !

Roy B • April 16, 2006 8:09 AM

Even is this were true, who amongst us but the totally paranoid really cares? I think the government has more important things to do than read our email. I feel more secure, thinking that they’re able to have computers scout for “key” words on calls inbound/outbound our country. You think they’re trying to catch you saying bad things about Bush? Get real!

Chris Walsh • April 16, 2006 10:05 AM

I’m more impressed by the seven 9s of reliability than by the speed. I’m not up on telco stuff, but a bit over three seconds of down-time per year is pretty good, even by carrier standards, isn’t it?

JakeS • April 16, 2006 10:53 AM

The question is not so much, how paranoid am I, as how paranoid are the NSA and the agencies that they pass information to?  We’ve already seen ridiculous actions justified in the name of the War on Terror – for example, innocent people are prevented from flying, or subjected to repeated delays and intensive searches, simply because they have the same name as someone on a secret no-fly list.  What innocent word or phrase might I use in an innocent e-mail, that their key-word scan would pick up and put me in line for harassment in the future?

Remember that saying bad things about the Bush adminstration makes you a liberal, unpatriotic, terrorist sympathizer.  That in itself won’t get you on the no-fly list, but it certainly gets you points in the NSA database.  Add a few more ‘negative’ factors:  simple, obvious ones like Arab origins, Muslim faith, a beard, being in touch with relations in certain countries…  or having a friend who’s any of the above…  or living near one, even if you don’t know him (that’s what got Jean-Charles de Menezes shot dead in London)…  and whoops, you’re over the top.

If you think that’s silly, imagine it reversed.  Bush has only two-and-a-half more years in office.  Will the mid-term elections change the balance in Congress?  Will your party’s candidate be elected in 2008?  Do you want this surveillance capability, and the power that it gives, to fall into in the hands of an administration in which you have no confidence?

Kent Brockman • April 16, 2006 2:56 PM

“You think they’re trying to catch you saying bad things about Bush? Get real!”

I, for one, welcome our new Constitution-ignoring overlords. I’d like to remind them that as a trusted TV personality I could be helpful in rounding up others…

Anonymous • April 16, 2006 8:08 PM

“Lt. Gen Paul vanRiper’s use of unconventional communications – relay runners, etc. … to which the administration said, “You can’t do that!” and reversed the Red Team’s achievements.”

Well, they would say that, wouldn’t they. You don’t expect them to say “We have chaps in bedsheets on the ground in Samara right now”.

Remember that a lot of what we know today about cryptanalysis in WWII wasn’t even declassified until the seventies.

alien • April 17, 2006 10:00 AM

@AG
The equipment doesn’t need to be specific. An optical tap on fiber is pretty generic.

@Pat Cahalan
Beyond comprehension by mere mortals? Hardly… unless the internet security community aren’t “mere mortals” 😉 Check out http://www.endace.com/ for commercially available hardware. The DAG cards are advertised as being able to do network monitoring on OC192/STM-64 and 10gigE pipes.

This isn’t magic, its just technology. As Bruce said, the only thing that is new information here is exactly how the NSA got the wholesale surveillance in place on AT&T’s backbone.

Cheburashka • April 17, 2006 10:01 AM

I have such conflicted feelings now – fear for privacy vs. “wow, that is one impressive feat of engineering.”

Anonymous • April 17, 2006 10:18 AM

@googleIsGoo?d:

“A whole bunch of analysts handcrafting queries can do wonders for semantics I’m sure…”

Yes, we could. Report to gate 7 for departure. Number 47, your time is up.

Pat Cahalan • April 17, 2006 10:31 AM

@ Alien

unless the internet security community aren’t “mere mortals” 😉

They’re not – the average citizen (mere mortal) doesn’t know what an OC-192 is, let alone what layer 7 monitoring of an OC-48 is 🙂

Tell them (the average citizen) that the NSA is installing machines that can monitor 32,000 phone calls simultaneously and they’ll start to understand that “We’re only monitoring the terrorists” is baloney.

Matthew X. Economou • April 17, 2006 1:01 PM

From the Wired article, I’m not certain AT&T is doing anything wrong, nor am I convinced AT&T is engaging in wholesale surveillance. If I was CISO at AT&T, I would doing my duty by designing and building a sophisticated, out-of-band network security monitoring system. If I had the requisite government connections, I would be happy to get a three letter agency’s help with vetting my staff. Convincing the NSA to help out was probably pretty easy, given the national importance of our communications infrastructure.

Matthew X. Economou • April 17, 2006 4:29 PM

Also, regarding Pat’s comments, I don’t think there will be any uproar over the public discovery of wholesale surveillance. In fact, I think most people will accept it as necessary for their safety and survival.

Pat Cahalan • April 18, 2006 4:35 PM

@ Matthew

I don’t think there will be any uproar over the public discovery of
wholesale surveillance.

You don’t consider the response to the outing of the NSA surveillance program in December to be “any uproar”? I think we’ve already had an “any uproar”.

ScaredPatriot • May 9, 2006 2:33 PM

“You think they’re trying to catch you saying bad things about Bush? Get real!”

I have a “friend” that (in email) said a few derrogatory (sp) things about “President” Bush and he was rewarded with a vist from the MIB.

Watch what you say folks. They ARE watching ALL of US.

3in1OIL • February 24, 2007 8:16 PM

“The only thing to fear is, fear itself”
Before the telephone was moris code through powered lines to be able to comunicate. “Peices of Eight” also “two bits” as trading with silver. Nine numbers created the path to “ma bell” and became born the beast whom creator became man in the development of another addiction by which we use as a domestic breakthrough in technology otherwise known as a devilish line “how convenant”. How convvenant for my family to have 15 flat screen monitors in our household” Soon will our next Hitler be able to apear on them, “fun for the whole family”. If I press *5 this will bring me grocery list to my door. And if I press #7 the flying saucer comes to the backyard and promises me the things I have dearly longed for a ferarri, a G5, or a spread on the oceanside, after receiving this spontaneously, all to find it was a deceiving lie, and the ship drops a poisonous chemical and I see my refection and sores begin to come out of my face as the days grow shorter and the earth is scorched by the sun, a vivid memory of why we hold on to the things we most cherish before we must leave this earth with a gift to give knowledge. And todays knowledge is man has enough technology to know what you dream at night when you sleep in your own bed with being connected to equiptment or device. After reading this I hope you say, There is no way, brother…

Yours Truly,
3in1OIL

my quote “my fathers father was a slave a master to whome he hath chosen for his future children to carry only the name given to the male gender carried forth, in honer of tradition, as to the cornerstone of thy tribe a new name is given to the chosen few. As many are called.

Ezight • August 24, 2007 8:24 AM

Go back to using CB Radio.

Yank out your hard drive and use LIVE CD’S
Like Puppy linux that runs straight from the CD.

Has opera,firefox,mp3 players,Runs straight fro mthe CD and leaves no trails on your HD since it requires no HARD DRIVE be installed in the computer to work.

Watch out at the borders ,don’t carry a laptop–they will take it from you..
Use a live CD instead and access your stuff from wifi and store your items online since you can’t store using puppy–or you can use a Sandisk cruiser–but–they may take that from you cause it’s (Storage) to you know who.

Nobody • May 11, 2008 5:35 PM

Use FRS radios instead.
Or CB radio.
Back in the 80’s when we were tinkering with The TRS 80’S we died not have the internet.
We simply made a phone call and shook hands directly with the macjhine we wanted to talk too.
Now days if we had to go back to that we could use 448 bit blowfish to swap info over the phone.
Can you imagine a 1024 bit blowfish encryption scheme running on Zterm or or something along the lines of a chat room program running on one of these fast dogs-AKA-2.4 Ghz machine?.

You could also set up your own encrypted FTP server on a wireless network.
Throw a USB dongle up on a 75 foor tower and see how far you get out.

Out west 2 years ago they had a wifi shoot out to see how far they could get a typical dongle to transmit. They got 14 miles out of it conected to one of those old dishes from the early 90’s in the desert.Very impressive!!!! to say the least.

john • April 2, 2016 6:00 PM

Mr. Schneier,

Some say this, and I quote:

“The 6400 also has the ability to block E-mails within an ISP such as AOL, without the Internet Service Provider actually even being aware that it is happening.

Based on how this computer is configured, it may well be capable of accomplishing just about any function on the Internet that its users deem necessary, including the deleting of Websites as well as the tampering of them, all without the Webhosts even realizing that this is happening to some of their customers’ accounts.”

Can you give us your expert opinion as to if any of this is possible? I
would imagine that none of this is possible because presumably the narus can
only, theoretically, data diddle with the mirrored copy of the fiber tap that
is redirected to the narus 6400, which then passes data on to the NSA,
and the other copy is presumably the one that used by the phone company switch, as usual, to meet their customers needs. Is that so?

j • April 2, 2016 6:35 PM

Mr. Schneier,

Some say this, and I quote:

“The 6400 also has the ability to block E-mails within an ISP such as AOL, without the Internet Service Provider actually even being aware that it is happening.

Based on how this computer is configured, it may well be capable of accomplishing just about any function on the Internet that its users deem necessary, including the deleting of Websites as well as the tampering of them, all without the Webhosts even realizing that this is happening to some of their customers’ accounts.”

Can you give us your expert opinion as to if any of this is possible? I
would imagine that none of this is possible because presumably the narus can
only, theoretically, data diddle with the mirrored copy of the fiber tap that
is redirected to the narus 6400, which then passes data on to the NSA,
and the other copy is presumably the one that used by the phone company switch, as usual.

However, supposing the Israeli designed chip has a backdoor, which never went
away just because Boeing is now selling it, would that not mean that the Israeli
mossad could data diddle with what is sent to the NSA by the narus box?

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.