Schneier on Security

Unfortunately, legislation, such the Patriot Act and others, expand the envelop of what can be considered "open sources" from a government/intelligence perspective. Will anyone notice the incremental erosion to privacy and personal liberty when their PC is declared a "public" information source?

Interesting article if you substitute "intelligence" with "marketing." I wonder what it would look like if such data mining efforts were used by governments to persuade and influence public attitudes in a particular direction, i.e. market the governments political agenda.

This is really a brilliant idea: reading newspapers and internet to understand what's going on in the world. Americans can be proud of their intelligence chief - it took a genius to come up with such an extraordinaryly clever plan. I only wonder why they haven't kept it secret - everybody will be busy to copy the idea.

Yeah, we are fortunate indeed to have supernaturally brilliant leaders. If _I_ suggested the newspaper-reading idea, folks might reasonably suppose I had read some history; our SBLs are entirely safe from any such suspicion.

There have been people pushing for this for a long time, and it's good to hear that it finally is becoming a reality (thought the implementation matters far more than the announcement). I saw an ex-CIA guy at HOPE 5 (or was it defcon?) rally on about this. It is definitely a good idea.

What exactly are "commercial databases"? To me that could include airline records, credit card and bank records, phone records. I know would conflict with the headline phrase "Open Source Center", but the devil is always in the details.

@Bruce

"How here's a good idea" - is that a typo or just sarcasm? :-)

They... they don't already do all that?

But, what if one arm of the government is planting false or misleading stories that this other arm of the government is reading without communicating that? Heck, the CIA might have actually been convinced to go to war in Iraq.

So will the stories planted by the US military 'information operations' troops in Iraqi newspapers be considered 'publicly available information' to inform US intelligence decisions?

I cannot imagine any government body admitting all its lies to any other government body.

Anyone who thinks this isn't done already is naive.

Anyone see Three Days of the Condor?

I'm a little fascinated and appalled to read this. It was at one of the earlier CFP conferences that I first heard this notion floated. Of course, in military circles 15 years from initial plan to deployment is actually pretty fast...

"'How here's a good idea' - is that a typo or just sarcasm?"

Fair question.

I actually think it is a good idea. From what I've been led to understood, the U.S. doesn't use open source intelligence nearly enough.

"Anyone who thinks this isn't done already is naive."

Yep, and if you go and read the original report, you'll discover that's what it says. The difference now, though is "that the center aims to spread its findings and expertise across the US intelligence community, which consists of 15 agencies."

So pretty much all of this thread has missed the point.

If this is truly public information, I would think it would be better to contract this out to a few different small companies. In order for this to be useful, those collecting and reporting on the information need to be able to change their methods quickly to keep up with the times. This is the whole idea of free markets - if someone comes up with a better way to collect and analyze information, they steal market share from the other companies providing inferior services. Governments are good at creating a rigid framework for the lowest common denominator. The most creative solutions rarely come out of government agencies.

They have been "reading newspapers" for years; the full-time job of most junior CIA analysts is to read foreign newspapers. What's new is the use of commercial marketing databases and other sources.

Maybe they should start reading this blog.

Since federal law restricts the government from keeping databases on private citizens, I'm wondering if they aren't going to be tapping into corporate assets from ChoicePoint and LexisNexis/Seisint.

"If this is truly public information, I would think it would be better to contract this out to a few different small companies."

That is already where open source inteliigence is coming from. Newspapers, LexisNexis, Factiva, Oxford Analytica, Jane's Information Group, etc. are all companies.

Sounds like TIA (Total Information Awareness) is back online.

I'd argue against it being a good idea, at least for the near future. The problem our intelligence agencies face right now is not the traditional problem of not being able to gather enough information; rather, the problem is that they gather so much information that they cannot possibly analyze even a small fraction of it. I would argue that we need to focus on filtering and analyzing data, not adding another firehose to the brigade already pounding our systems.

"...the problem is that they gather so much information that they cannot possibly analyze even a small fraction of it."

But much of what is available in open source is analyzed. There's a whole lot of intelligence analysis being done for business purposes. I'd like the CIA to tap that, if for no other reason than a sanity check on their classified sources.

"Since federal law restricts the government from keeping databases on private citizens, I'm wondering if they aren't going to be tapping into corporate assets from ChoicePoint and LexisNexis/Seisint."

They already are. That has nothing to do with this, but they already are. There's an unholy alliance between commercial data brokers and the government that is threatening to undermine a lot of our personal privacy.

An interesting article on Open Source Intel at the CIA

http://www.cia.gov/csi/studies/Vol49no2/reexamining_the_distinction_3.htm

Okay this is more entertainment than is safe for me to have at work :-) Makes me wonder if there are established correlation or cross-reference logic algorithms to test the likely veracity of information and assign it something akin to a credibility rating ... the amount of caffeine in my system might have prevented that sentence from making any sense.

In a nation where, allegedly, you can be prosecuted for espionage for tearing out a page of a US phone book and giving it to The Wrong Foreigner (I always wondered about that), I'm shocked that it has taken our intelligence community this long to come up with something like this. Or has it just taken this long for them to announce it?

Also, the FBIS are studs, everyone one. Period. I <3 those people.

Above comment re: phone books and FBIS was me, forgot to sign.

"Since federal law restricts the government from keeping databases on private citizens, ...."

Huh? What Federal law does that? And what about the IRS, the various motor and licensing registries, land title registries, voter registration, criminal records, ....

"rather, the problem is that they gather so much information that they cannot possibly analyze even a small fraction of it. I would argue that we need to focus on filtering and analyzing data, not adding another firehose to the brigade already pounding our systems."

Ahh but in essence this is the start of a filtering operation. The first aim is to cut down on duplication of effort. The second aim should be the allocation of resources other groups were using previously (that duplicated said effort) which can now go into filtering and interpreting the information that is relevant to the appropriate organisation.

I would hazard a guess that the CIA-managed group doing this will probably tag the information in various ways to assist in filtering, but other groups could use or ignore that information at their whim. However, if the tagging effort shows who the CIA-managed group would think would be most interested in this information, and it turns out that this information is actually disinformation spread by another group, said group spreading the disinformation could inform the groups they feel it is appropriate to tell them that it *is* disinformation. This in itself could be invaluable to the intelligence community as a whole.

Wasn't the similar position described in the 1975 film Three Days of the Condor ?
It was mentioned eariler by Pat Cahalan.

I think this unclassified article on the CIA website from earlier this year helps explain some background to this:
http://www.cia.gov/csi/studies/Vol49no2/reexamining_the_distinction_3.htm

Years ago the US had a bit of a spat with New Zealand. They announced that they had no intelligence to share with NZ.

The New Zealanders laughed because they had known for a long time that the US had no intelligence...

This is just further proof that they were right.

The US Intelligence community have been poor at exploiting open source intelligence. They tend to favour secret intelligence over all source. The end result according to Open Source intelligence guru Robert Steele is when a question is asked:

"CIA tells you either that they don't know, or what they know is too secret to tell you."

A nice article on Open Source Intelligence by John Parry Barlow can be found here:

http://www.forbes.com/asap/2002/1007/042.html

Plus for those interested the

www.oss.net website has a lot of links.

The Able Danger saga [http://www.gsnmagazine.com/sep_05/shaffer_interview.html] is also relevant.

Thank you to Felix for noting that OSS.Net has been doing this. I founded the USMC Intelligence Center in 1988, and was shocked to discover, after a career as a spy, that 80% of what I needed to do policy, acquisition, operational, and logistics intelligence was not secret, not in English, not online, and not known to anyone in DC. When I first proposed an alternative paradigm for national intelligence (E3i: Ethics, Ecology, Evolution, and Intellignece) in Whole Earth Review (Fall 1992), I was called a lunatic by the leadership of the CIA, which I happily left for the saner USMC. Lunatic. They have wasted 17 years because of their stupid corruption (military-industrial complex) and stupid mind-sets (not invented here, we only do secrets). Their current initiatives are "lipstick on the pig" (see picture of the pig at www.oss.net, Open Source Agency portal page). The good news is that a good person is being put in charge on the IC side, above the losers in the Foreign Broadcast Information Service, we have a Congressional initiative to create a Congressional Intelligence Office like the CRS and CBO but focused on providing Public Daily Briefs and Public Net Assessments to all, and I have a multi-billion dollar initiative that can be summed up, across 20 companies, as "The Googlization of Intelligence." See the brief at www.oss.net/Hackers. I have conference coming up, IOP '06 in DC 16-20 Jan 06, and I want to create some buzz--anyone who sends in a registration from with the name SCHNEIER in the upper right hand corner can pay half price. See conference details at www.oss.net/IOP. There is indeed a "Collective Intelligence" revolution going on, and this is one of the streams feeding power to the people. It will be tough. Wall Street is starting to fear us as they see the days of undisturbed global looting coming to an end.

ah yes...Robert David Steele...that's the guy who I saw talk at HOPE or wherever (I recognize the phrase "was not secret, not in English, not online, and not known to anyone in DC"

:P

This probably will make the CIA smarter. So here's a question: if the CIA is going to function by reading open sources and posting a bunch of blogs...aren't all of us here doing the same thing? Is there any reason for us to trust official government pronouncements of what their analysts say, over our own conclusions from reading a bunch of blogs? Particularly given their track record over the last several years?

Now, a really good idea would be to do all this, plus put some kind of idea futures market in place, like DARPA suggested a couple years ago. A good book on this and related subjects is The Wisdom of Crowds....if you have a bunch of people making independent judgements (no groupthink), and some way of aggregating those judgements, you can get a level of intelligence that exceeds that of the individuals involved. For example, the U.S. Navy found a lost submarine by aggregating the individual estimates of a bunch of disparate experts...the aggregate coordinates were a lot closer to where the sub was found than any individual's estimate.

What a good idea.
Why the heck didn't they do this before? This annoucement means the CIA, inexusably, hadn't a good working plan to encorporate information from the public sphere into its general analysis of intelligence, despite this information being a more reliable source of information compared to spies and secret information, and despite such a plan to analyse public information being more easily testable (to find its flaws).

At least Negroponte's somewhat candid about it.

Plainly, while lies and dis-information can permeate the public sphere, there are, generally, counterarguments to those myths and lies also in the public sphere, and an astute fud-cutter like a cia agent could make use of this information, without, necessarily, understanding all the details. This kind of honest analysis information of public information is more reliable, I think, that the secret sphere, where the process of verification is ad-hoc, or, evidently, just not done even attempted (and sometime it just cannot be verified).

Recently, I think the CIA has (obviously) placed too much faith on the information from covert operatives simply because the information was covert, and ignored (as an organization) the information from the public realm that could cast doubt upon secret intelligence. Info from 'covert' sources went straight to the top and formed the basis of policy. Info from drudging fact checkers and skeptical thinkers was infinitely filtered, and never made it to the top in a coherent way. At least, that's the 'official line', and it fits well with what's known of the missed opportunities to maybe try to prevent some of 9/11, and perhaps have avoided some of the crap that was spread leading up to the war in iraq, and perhaps would have prevented some of the odd policies and decisions by the US during the initial active combat operations in iraq, and subsequently.

Probably there were as well more political, plain sickening, noble, or incompetent reasons for the initial war policy justifications. But a prudent analysis of how government used intelligence to form policy and the methods of gathering intelligence could have identified the potential for failure before it manifested itself (all these processes are just the basic role for a government, in any situation, not just intelligence). If the intelligence agencies were set up properly, government wouldn't have allowed the tenuous rationales for the iraq war to be so easily made, and so easily got away with publicly conflating the noble, political, incompetent, (etc) reasons. There wouldn't have needed to be such a political 9/11 commision, since the government would have been able to say straight away what was possible to know, and what was possible to prevent. This was foreseeable well before the war, and before 9/11, and to both government and the public, and certainly the movers of both major political parties, whoever was president.

-----

I hope Negroponte's predecessors are asked hard questions about why this is 'new' policy.

Does anyone remember the Community Open Source Program that was at FBIS, before FBIS became the Open Source Center?

I do...

What, CIA is going to create another COSPO?

Open Source collection has been going on for ages. Go search up "Robert Steele" - he's been banging the drum forever.

It's interesting - Negroponte is announcing old news as if it's interesting.