Total Information Awareness Is Back

Remember Total Information Awareness?

In November 2002, the New York Times reported that the Defense Advanced Research Projects Agency (DARPA) was developing a tracking system called “Total Information Awareness” (TIA), which was intended to detect terrorists through analyzing troves of information. The system, developed under the direction of John Poindexter, then-director of DARPA’s Information Awareness Office, was envisioned to give law enforcement access to private data without suspicion of wrongdoing or a warrant.

TIA purported to capture the “information signature” of people so that the government could track potential terrorists and criminals involved in “low-intensity/low-density” forms of warfare and crime. The goal was to track individuals through collecting as much information about them as possible and using computer algorithms and human analysis to detect potential activity.

The project called for the development of “revolutionary technology for ultra-large all-source information repositories,” which would contain information from multiple sources to create a “virtual, centralized, grand database.” This database would be populated by transaction data contained in current databases such as financial records, medical records, communication records, and travel records as well as new sources of information. Also fed into the database would be intelligence data.

The public found it so abhorrent, and objected so forcefully, that Congress killed funding for the program in September 2003.

None of us thought that meant the end of TIA, only that it would turn into a classified program and be renamed. Well, the program is now called Tangram, and it is classified:

The government’s top intelligence agency is building a computerized system to search very large stores of information for patterns of activity that look like terrorist planning. The system, which is run by the Office of the Director of National Intelligence, is in the early research phases and is being tested, in part, with government intelligence that may contain information on U.S. citizens and other people inside the country.

It encompasses existing profiling and detection systems, including those that create “suspicion scores” for suspected terrorists by analyzing very large databases of government intelligence, as well as records of individuals’ private communications, financial transactions, and other everyday activities.

The information about Tangram comes from a government document looking for contractors to help design and build the system.

DefenseTech writes:

The document, which is a description of the Tangram program for potential contractors, describes other, existing profiling and detection systems that haven’t moved beyond so-called “guilt-by-association models,” which link suspected terrorists to potential associates, but apparently don’t tell analysts much about why those links are significant. Tangram wants to improve upon these methods, as well as investigate the effectiveness of other detection links such as “collective inferencing,” which attempt to create suspicion scores of entire networks of people simultaneously.

Data mining for terrorists has always been a dumb idea. And the existence of Tangram illustrates the problem with Congress trying to stop a program by killing its funding; it just comes back under a different name.

Tags: , , , , , , , , , ,

Posted on October 31, 2006 at 6:59 AM • 33 Comments

Comments

Anonymous • October 31, 2006 7:52 AM

I am sorry for those of you who live in the US, but I believe it is already too late to stop this avalanche of a totalitarian power grab. You have already lost, and probably forever so.

Curiosity • October 31, 2006 8:09 AM

Why the Air Force?

Also, a total budget of $49M for 5 years? That’s barely enough to build and run a good corporate email system.

I guarantee this: With such lofty goals and such a small budget, and with the work being done by government consultants (by definition, those who are not good enough to work on for-profit systems), whatever they end up building would be completely useless, even if the original goals made any sense (and I agree with Bruce that they do not).

Unfortunately, such incompetence won’t make this any less damaging to the freedom of America’s citizens.

We need more hay! • October 31, 2006 8:23 AM

@Curiosity

Exactly. None of these programs are really for what they claim to be for – because if they were truly legitimate, the government would not need to hide them behind the classification curtain and flout the will of the general public.

They have two purposes:

1) To roll a lot of money into certain corporate and governmental pockets

2) To allow certain corporate and governmental entities to gather information that will allow that money to continue to roll into certain corporate and governmental pockets.

If this was really about catching Bin Laden, it would have been done by now. By flouting the will of the American people and re-implementing this project, it is clear that this action is hardly in their best interest.

Antonin • October 31, 2006 8:43 AM

And the existence of Tangram illustrates the problem with Congress trying to stop a program by killing its funding; it just comes back under a different name.

Did they actually intend to stop it, though? Maybe me hat contains a bit too much tinfoil today, but I wouldn’t be very surprised if the whole thing was just a (successful!) attempt to mollify the general public and if Congress full well knew that the program wouldn’t actually be stopped.

I’m aware of Hanlon’s Razor, of course, but since this is politics on a high level, I’m not sure whether it’s actually applicable – or whether the opposite (“never attribute to stupidity that which can adequately be explained by malice”) might actually hold true. (Maybe that’s a bit too cynical, but that’s hard to say, too.)

aikimark • October 31, 2006 9:41 AM

Tangram is the Camel’s Nose

Lesson to be learned:
If your tent isn’t a den of terrorism, extremism, or criminal planning/activity then you don’t have to worry about the Camel’s Nose sniffing around.

Roy • October 31, 2006 10:04 AM

We can only hope the terrorists have not yet heard of handwriting — that dastardly means of communication which is utterly invisible to electronic eavesdropping. It is said that a handwritten message can be sent unseen across the globe reliably and cheaply inside a paper envelope, guided by inky figures and a colored stamp on the front side.

foolishly idealistic • October 31, 2006 10:37 AM

It seems to me that no matter how hard we try to figure out what the terrorists are doing we will always be playing catch up. We can dump trillions of dollars into hyper advanced data mining programs, we can rob every last citizen of every last right and the terrorists will just keep coming. In fact, we will probably just end up making terrorists of our citizens.

We have the wrong mind set. We are trying to figure out what people are going to do before they do it instead of trying to figure out what we need to do to make them not want to do it.

The only way to stop the terrorists is to stop making an environment where fanatical extremism can thrive. To do this we would probably have to sacrifice some of our economic strength and global political clout, but wouldn’t a safer world be worth it?

Before the inevitable reply gets posted about being foolishly idealistic and living in a fantasy world, yes I know I am. And I wish there were more people like me. Maybe then people wouldn’t want to blow us up all the time.

X the Unknown • October 31, 2006 10:39 AM

@aikimark “f your tent isn’t a den of terrorism, extremism, or criminal planning/activity then you don’t have to worry about the Camel’s Nose sniffing around.”

Of course, exactly what constitutes “criminal activity” – or more importantly, what will constitute retroactive criminal activity – is the whole problem. What with the president able to declare citizens “enemy combatants” based on unreviewed and classified parameters, and able to circumvent both the Constitution and the Geneva Conventions, I’m pretty sure I don’t want that camel anywhere near me.

Carlo Graziani • October 31, 2006 11:07 AM

Obviously, the NSA was having so much fun DOS-ing the FBI with fecal-quality “leads” generated from data-mining of phone records, it stands to reason the rest of DOD would want to get in the act as well.

God have mercy on the person who gets hit with the — now inevitable — 5-sigma false-positive trigger. The Feds will have none.

Stacktrace • October 31, 2006 11:53 AM

One way this database might backfire on the government is this: Suppose they will indeed get zillions of false positives, forcing them to sieve vigorously, following only a small fraction of the “leads”. Now if another terrorist attack comes along which isn’t prevented, the equivalent of the 9/11 commission will have somebody look up the attackers in the Tangram database. If those entries look “suspicious” to a human analyst, the government will be blamed for missing warning signs as for 9/11.

joppy • October 31, 2006 11:57 AM

RonK: I heard a rumour lately that some people do meet in person. For example, students who gather for learning, sports club members training for fun and fitness, people walking their dog, or asking each other for the time of day in crowded and uncrowded places.

I bet traffic analysis, no matter how clever it tries to be, will not identify the connection between four men who plan something evil. Possibly not even 20 who use messengers, snail mail, amateur radio, extra whitespace in web pages or spam, or even email or chat across continents.

Opening letters or breaking encryprion won’t help much, too. It is fairly easy to negotiate a simple second meaning of innocent words. For example, “Hey Susi, today my dog pissed on the carpet” could mean “Put the anthrax into ground water on thursday.”

By the way, I agree with the poster “We need more hay!” that much governmental spending, possibly including this TIA/Tangram thingy, is corruptly abused by powerful and rich people to accumulate even more power and money.

I also agree with “foolishly”, and guess that the corruption and infinite greed described in the previous paragraph is one of the reasons that make people hate the USA.

I have a subtle feeling that the term “nonbeliever” does not always mean someone who does not believe in and pray to Allah, but sometimes in an abstract sense someone who does not respect the spirit behind most religions that demand modesty, honesty, and respect for others. But it requires strong words and simple widely recognized symbols to unite people for resistance.

Don’t get me wrong. Harming innocents is always wrong, terror is wrong, and of course there must be countermeasures like there is against other forms of crime.

But people do not blow themselves up unless there is something so unfair that it drives them nuts. No smoke without fire. Making smoke illegal does not help much when the house burns.

Probitas • October 31, 2006 12:03 PM

“Tangram is the Camel’s Nose”

The other lesson to be learned is that Camel’s Noses rarely exist independant of camels, which are notoriously stubborn and difficult to train. Although I have no first hand experience with the beasts, I am guessing that some camels, upon entering any given tent “nose deep”, might find it impossible to resist the urge to enter further.

So the question is, can a government be trusted to be more evolved than a camel? Judging from the current state of the campaigns in my state, the answer is a resounding “No”.

Fenris Fox • October 31, 2006 12:12 PM

@Roy

“We can only hope the terrorists have not yet heard of handwriting — that dastardly means of communication which is utterly invisible to electronic eavesdropping. It is said that a handwritten message can be sent unseen across the globe reliably and cheaply inside a paper envelope, guided by inky figures and a colored stamp on the front side.”

Or strong crypto, or steganography.. but your idea needs a lot less technological expertise.

(And yes, I know the steganography thing with the terrorists was supposedly an urban legend.. but still.. it could happen.

My question is.. can’t the U.S. Military detect the heat/exhaust from Osama’s generators [to run his cryptographic platforms] in those cold mountains? =;o) [sarcasm..])

Fenris Fox • October 31, 2006 12:22 PM

@joppy

“Opening letters or breaking encryprion won’t help much, too. It is fairly easy to negotiate a simple second meaning of innocent words. For example, “Hey Susi, today my dog pissed on the carpet” could mean “Put the anthrax into ground water on thursday.””

And this is one of the most effective means of secret communication, to this day. AFAIK, unlike (at least in theory, given the way some modern algorithms are) ciphers, there’s no way to break such a code by cryptanalysis. The only effective means are: 1.) Find a codebook (and the terrorists likely just keep it in their head), or 2.) Interrogate/torture the codes out of a suspect.

(NOTE: This is NOT an accusation of the U.S. torturing detainees. However, some governments may be more than happy to resort to such barbarism.)

MS • October 31, 2006 1:28 PM

@ Fenris Fox
“Or strong crypto, or steganography.. but your idea needs a lot less technological expertise.”

Can you be sure – so sure you’d bet your life on it – that Totalitarian Sam hasn’t forced a back door into all windows machines, or could do so in very short order, either as part of the boot process call to the mother ship or via windows update? Since strong crypto and steg rely on the ongoing integrity of your OS, does that not render them as good as useless against him? Sure, you could run as a standalone machine and send all communications by printed letter, but that is slow and as snailmail volumes drop and prices likely rise creating a vicious spiral, we may well reach a point where snail mail becomes so rare that even using it becomes grounds for suspicion. Furthermore, it is not difficult to imagine a time in the not so distant future where standalone machines also become obsolete or so rare that the skills required to build/use one limit them to a tiny, highly skilled group, with the result that, again, mere use becomes grounds for suspicion.

I don’t know what the situation is with Linux. Is it any different? Is there any OS of whose integrity one can be sure? If not, it seems eternal vigilance and open source – of both software AND hardware – are our best hope.

linuxguy • October 31, 2006 2:11 PM

If you are looking for an OS with integrity, try OpenBSD. http://www.openbsd.org “Only one remote hole in the default install, in more than 10 years!” and
“The OpenBSD project produces a FREE, multi-platform 4.4BSD-based UNIX-like operating system.” NetBSD and FreeBSD also come to mind, but OpenBSD is probably the preferred OS among the really security conscious. There are “hardened” versions of various Linux distributions as well.

for_real • October 31, 2006 2:14 PM

Substitute “opposition” for “terrorist” and you begin to understand what this is about.
Certainly this item alone is enough to put me on a list, but since I’m already on it …

X the Unknown • October 31, 2006 3:01 PM

@Fenris Fox: “My question is.. can’t the U.S. Military detect the heat/exhaust from Osama’s generators [to run his cryptographic platforms] in those cold mountains? =;o) [sarcasm..])”

Clearly, if the US REALLY wanted Osama Bin Laden, they could have earmarked 1 Billion (heck, even 10 Billion) for a reward, saving hundreds of Billions, and getting results by now. A reward that looks like the GNP of small countries is likely to put a real strain on the trust and structure of the cell system and heirarchy even of a group of religious extremists.

malcontent • October 31, 2006 3:24 PM

The real problem with this program is that the “terrorist” are those of us on the progressive, or even libertarian, or legitimately conservative side of the fence. Anyone who’s not a complete fascist asshole will be collected and catagorized by this system. If ever anyone starts to think that the government in this country has gotten completely out of hand and decides it’s time to rebel, they’ll be snuffed out before they even have a change to take down more than a handful of fascists.

Bob • October 31, 2006 7:16 PM

Congress’ problem is not that the program can just re-surface under a different name. The bill killing the funding explicitly prohibited the program under any name and prohibited any use of funding for any similar program.

The problem is that there is a constitutional crisis underway where one branch of the government is trying a replay of Watergate (same actors) and firmly asserts executive privilege over congress, brushing off any checks and balances. Just recently an “administration official” stated that the response to any subpoenas (out of an increasingly likely Dem congress) would be a fight on every issue, every time all the way to the Supreme Court. They are thumbing their nose at Congress and Congress is bending over.

Unless this unitary executive vision is nipped you can kiss the grand experiment of this republic goodbye.

SpannerITWks • November 1, 2006 2:47 AM

How secure would you expect a ” supposedly ” secure site to be, that’s connected with all this ? You know a .MIL/GOV www that states it requires ID/Certs etc before being able to proceed any further and download ” secret ” docs. Well see if you can download any docs from the following www –

https://extranet.if.afrl.af.mil/programs/TANGRAM/TANGRAM-extra.html

Not saying that i have of course !

As for why the Air Force, well it’s not just them, but the Navy and Army etc too, that are involved either directly and/or indirectly with All sorts of things, and i mean ALL sorts. Always have been as well.

Spanner

Rob Mayfield • November 1, 2006 4:36 AM

At least it distracts some of them briefly from the “Total Entertainment Combat Video Game” (thanks Roger, see you on Feb 1 😉 ).

sam paine • November 2, 2006 2:16 PM

this is a surprise to any of you? kiss the grand experiment good-bye? we already did- 9-11 was the opening of an endgame that had been planned for years- recent legislation gave the president authority over all armed forces on US soil- and Americans puzzle and debate overwhich corporate-sponsored candidate to elect- check – next they’ll come for our guns- checkmate

Luther R. Norman • November 2, 2006 10:54 PM

Did you know that the Bush administration blocked announcing the Wanta plan? This was a good sum of money that was to be used to help the American taxpayer (about 24 trillion dollars or so), drowning in the morass of the Iraq War.

Actually, according to reliable sources, Bush planned to steal the funds and split them among is cohorts (this could have been more like 75 trillion dollars!).

No one learned about it in time because the story was Mel Gibson’s remarks while drunk!

Ceri • June 26, 2010 6:23 AM

What’s depressing is people like myself with a legitimate reason to look up information on compounds that have been used in IEDs get flagged on all the data miners for possible terrorist ties.

Oh and the alternate phrase cipher is not uncrackable for some experienced cryptanalysts provided they have sufficient material that is known to possess a cipher. It always amazed me the Germans didn’t seem to possess a good enough crypto team to crack the radio broadcasts to the French resistance and insurgency teams during WWII “John has a red mustache” might seem unbreakable, but eventually patterns emerge with analysis that allow for the breaking of the cipher, we use this in testing wifi security today to break WEP passphrases, it’s called statistical analysis, something we use for ALL substitution ciphers with great success and a code phrase is still merely a substitution cipher, read David Kahn’s books sometime, the Code Breakers is a particular favourite of mine as the write up on the breaking of Enigma points out the dangers of using even the most simple repetition in a otherwise phenomenally secure (for the time) cipher system. Asymmetric Encryption improves security but the old fashion methods can still be employed to break it eventually (yes I know the thousand of years argument but look at the write up on AES).

I think our greatest hazard in the fight for protecting our own privacy is the fact the governments can request whatever they like of a service provider and be assured sooner or later they’ll get it, you toast your servers they imprison you, you keep backups they want it, you keep logs they’ll have them, you don’t they’ll demand you do… I’m not even being paranoid here, the government here have been trying to get ISPs to track all data movement for over 10 years, when my ISP at the time tried to comply it trashed their servers just trying to log all the DIALUP traffic.

If they want us to behave they’d be better off leaving us alone, we get up to less mischief when we’re not being watched. Anyone remember the reported trashing of Echelon by hackers deliberately posting terrorist related emails?

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.