Opinion Monitoring Software

Interesting research:

A consortium of major universities, using Homeland Security Department money, is developing software that would let the government monitor negative opinions of the United States or its leaders in newspapers and other publications overseas.

Such a “sentiment analysis” is intended to identify potential threats to the nation, security officials said.

This kind of thing could actually be a good idea. For example, it could be used to help the administration understand how we are viewed by people in other countries, and make us more responsible players on the world stage as a result.

On the other hand, this kind of thing could also be used to track critics of the U.S., and to aid in media manipulation. It is not unusual for government leaders to punish reporters who do not provide favorable coverage by excluding them from important events and key briefings, and this could facilitate that. At the very least, it would have a chilling effect on worldwide freedom of the press.

Note also that the project director says that the system would not extend to domestic news sources:

It could take several years for such a monitoring system to be in place, said Joe Kielman, coordinator of the research effort. The monitoring would not extend to United States news, Mr. Kielman said.

But a few paragraphs later:

The articles in the database include work from many American newspapers and news wire services, including The Miami Herald and The New York Times, as well as foreign sources like Agence France-Presse and The Dawn, a newspaper in Pakistan.

I have to admit I find the whole thing a bit too Orwellian for my tastes.

Tags: ,

Posted on October 6, 2006 at 11:57 AM • 50 Comments

Comments

Paulie • October 6, 2006 12:15 PM

In Frank Herbert’s “Dosadi Experiment”, he has a plot device called the “Demopol”.

“The DemoPol was always held up to us as the ultimate equalizer, a source of decision-making mircales. It was supposed to produce a growing body of knowledge about what a society really needed. It was thought to produce justice in all cases despite any odds.??? (p. 98)

Roy • October 6, 2006 12:17 PM

One twist is that the monitoring would be unable to tell authentic news from planted news, so when the US government plants favorable stories in foreign papers, the White House will be swayed by its own propaganda — and that can’t be good.

I.P. Daily • October 6, 2006 12:21 PM

@Roy
Roy, I doubt that the US government would have any problem sorting out planted “stories” or opinions. They are far too intelligent.

Joe • October 6, 2006 12:24 PM

I’d like to know if it can detect irony–eg, satire. I’ve seen some articles claiming that the Daily Show is just as substantive as straight news. It would be interesting to see if the “sentiment analysis” of Stephen Colbert showed favorable sentiment for the Administration.

Kevin Davidson • October 6, 2006 12:24 PM

“This … could be used to help the administration understand how we are viewed by people in other countries, and make us more responsible players on the world stage as a result.”

Not THIS administration. THIS administration is in denial.

Geoff Lane • October 6, 2006 12:47 PM

Where are they going to find people with the necessary language skills?

Or are they going to rely on machine translations..

I notice that Google has added Arabic to thir translation services.

t3knomanser • October 6, 2006 1:02 PM

Am I the only one that doesn’t think this is Orwellian? It’s like making an industrial version of “Google my name and see what comes up.”

Antonin • October 6, 2006 1:03 PM

From TFA:

“There has to be guidelines and restrictions on the use of this kind of technology by the government,??? Professor Wiebe said. “But it doesn’t mean it is not useful. It can just as easily help the government understand what is going on in places around the world.???

Ah, yes, the old “it’s not our technology that’s evil, it’s how it’s used” excuse. The Nuremberg defense of scientists…

Seriously, do they actually believe that a system like this will not be abused? If yes, then they’re either incredibly naive or purposefully turning a blind eye; and if no, then they’re lying about their true motives.

I’m not sure which it is, but neither option makes me feel good.

jayh • October 6, 2006 1:05 PM

It’s not, as described, particularly Orwellian, but is bizarre from an administration openly proud of the fact that world opinion is not considered important in policy decisions.

aslam • October 6, 2006 1:22 PM

Putting aside all the “Orwellian” possibilities, this seems like a waste of Homeland Security money, though it might be good for NLP technology. Tracking foreign opinion is a good idea, but expecting this effort to lead to info about specific threats is wishful thinking at best. Of course, if the intention is to find reasons to demonize individuals or countries, or drum up business for the Extraordinary Rendition industry, then this could be our ticket.

Swiss Connection • October 6, 2006 1:24 PM

There already is monitoring system on overseas opinion on the US. It is call the Stock Exchange.

Valdis Kletnieks • October 6, 2006 1:30 PM

My favorite part of the article:

“Researchers at institutions including Cornell, the University of Pittsburgh and the University of Utah intend to test the system on hundreds of articles published in 2001 and 2002 of topics like President Bush’s use of the term ‘axis of evil’, the handling of detainees at Guantánamo Bay, the debate over global warming and the coup attempt against President Hugo Chávez of Venezuela.”

Nothing like testing the system against known threats we face. If it can’t flag those, it’s not going to work on finding threats that aren’t self-inflicted….

Andre LePlume • October 6, 2006 1:45 PM

Let Google help:

“US foreign policy rules”: 8 results
“US foreign policy sucks”: 111 results

piglet • October 6, 2006 2:05 PM

“This kind of thing could actually be a good idea. For example, it could be used to help the administration understand how we are viewed by people in other countries, and make us more responsible players on the world stage as a result.”

As if you needed an expensive software to know world opinion about Bush and his gang of crooks.

Antonin • October 6, 2006 2:08 PM

@t3knomanser: Yes. Google News aggregates news stories, but it does not attempt to extract or aggregate opinions.

piglet • October 6, 2006 2:44 PM

“Is there any greater room for abuse here than say, Google News?”

If there weren’t, they could just use Google News, couldn’t they 😉

gurkhali • October 6, 2006 3:02 PM

Isn’t this why we have embassies in the first place? It’s too bad that good old fashioned diplomacy is being replaced by cumbersome and buggy software.

Valdis Kletnieks • October 6, 2006 3:07 PM

@gurkhali:

No, embassies are so we can feed the official lies our government wants their government to hear, and to receive the official lies their government wants us to hear. This is so that we can pick up the official lies that are appearing in the media courtesy of the country’s propaganda machine.

LazySumo • October 6, 2006 4:14 PM

“At the very least, it would have a chilling effect on worldwide freedom of the press.”

Right, as opposed to burning embassies because you don’t like a cartoon, THAT did wonders for worldwide freedom of the press.

t3knomanser • October 6, 2006 4:27 PM

@Antonin: But the reader does. Do you, perhaps, imagine that nobody in the government reads Google News?

I dunno, it seems to me that we’re just automating something that seems like a rational action to me anyway. It is in any organization’s best interests to keep tabs of what people think about said organization. If you can automate that process, thus freeing people up for useful work (well, different work, this is the government after all), I say have at.

theprez98 • October 6, 2006 5:14 PM

@jayh:

“an administration openly proud of the fact that world opinion is not considered important in policy decisions.”

Considering the world doesn’t get to vote in our presidential elections, I for one am glad world opinion doesn’t mean much.

@Kevin Davidson

“Not THIS administration. THIS administration is in denial.”

It’s nice to see people can find a reason in every post (however non-political it may be) to attack the Bush administration.

Alan • October 6, 2006 5:23 PM

Is it just me or are we seeing more and more approaches that are based on dowsing or mysticism than an actual formulated approach. Seems like every scammer out there has some new tool that will help find out who hates us and who likes us. Something that will magically find the needle in the haystack.

And the suckers keep buying…

What is the term for “Government by the Insane”?

Stefan Wagner • October 6, 2006 9:15 PM

Assuming the software will work, detect irony, and so on.
The most abstract usage I can imagine is a map, where every country is colored grey from white to black, indicating good or bad opinions.

The goverment may send the marines to suddenly black-turning countries without asking why.

Moving a mouse over the country could popup a balloontip, indicating the most cited reason like ‘The US put our president to the no-fly list’.

This would allow to elect less intelligent and diplomatic presidents in the US than today. Strictly speaking, this is a step in eliminating human presidents at all.

Davi Ottenheimer • October 7, 2006 12:59 AM

…and thus became the latest technology in the war on terror: Opinion-Guided Missles

Sources close to the Pentagon were quoted as saying “We just couldn’t trust our laser and heat seeking systems anymore to root out critics, sorry, I mean threats to our administration. Let’s face it, tank plinking a T-72 with a GBU-12 is one thing, but with these new OGMs we can take out a whole editorial team without any collateral damage to the sports section.”

Davi Ottenheimer • October 7, 2006 1:09 AM

“Not THIS administration. THIS administration is in denial.”

Indeed, this might be the crux of the research. Someone is perhaps thinking “After all we have done, their hatred of us just does not compute. We need to build a better system that can find the source of hate”.

Managers who ignore the input and advice of those around them often spend a fortune on new inputs of information, such as consultants who interview those around the management and provide a fresh copy of the same statements.

So on the whole this could be a good system if it allows those in charge of Homeland Security to become more receptive to information already available but hard for them to acknowledge.

On the other hand, I also worry about it being misused as a way to more accurately judge and therefore shoot the messengers.

Foil Hat • October 7, 2006 7:12 AM

I feel vindicated. A while back, I stopped using the same name for blog comments or anything else I do online. I can forsee a situation in the future were a tourist applying for a visa to go to the US is denied because of something they posted online years ago that they had forgot about.

@Antonin
IMHO, with time, this will be much worse than Google. Eventually, we may see systems that automatically associates data from multiple sources to build profiles of people.

Without wishing to sound paranoid, these days I try to stay off as many databases (retail, government or whatever) as possible so that I may enjoy some anonymity in the future.

Dscho • October 7, 2006 5:21 PM

@all answerers to Antonin

It is utterly naive to assume that this program will not be abused on a personal level. Rather, bloggers will be found who are critics of the Bush government, and will be punished.

Google News is harmless. It really shows the opinions in other states.

But you do not really want to admit to believe that the Bush government — the same government who insists that only surveillance not granted by an independent court may save the country! — is interested in the opinion of non-US citizens. Now, that would be naive. They’re no voters. Better to scare voters into voting for the GOP.

another_bruce • October 8, 2006 1:27 AM

opinion monitoring software?
i thought they already had this.
ok.
george w. bush blows chunks!
there, that’s as good as a diebold vote.

Davi Ottenheimer • October 9, 2006 11:49 AM

In related news, did you see the quote by the guy at the Iraqi Media Network?

“It is the right of the Iraqi government, as it combats terrorism, to silence any voice that tries to harm the national unity.”

More info here:
http://www.iht.com/articles/2006/09/29/africa/web.0926journo.php

“Under a broad new set of laws criminalizing speech that ridicules the government or its officials, some resurrected verbatim from Saddam Hussein’s penal code, roughly a dozen Iraqi journalists have been charged with offending public officials in the past year.”

averros • October 9, 2006 2:48 PM

Anyone who thinks that the government officials are in denial, are in denial of the fact that government officials never ever had serving interests of “the people” as their goal.

If you stop considering them working for the good of the Nation (or insert whatever high and noble evdeavours you think they’re supposed to be pursuing) and start thinking them as just people pursuing their own interests, the politics becomes quite comprehensible.

Davi Ottenheimer • October 9, 2006 11:11 PM

@ averros

Perhaps what you say is true for some people in most competitive endeavors. There are those who usually play by the rules to achieve the best they can while others are obsessed with loopholes and consolidation of power to neuter the genuine achievements of others. I would no sooner say all politicians are just people pursuing their own interests as I would say all professional athletes are doping.

http://bjsm.bmjjournals.com/cgi/content/full/37/4/335

In other words, given your blanket generalization, what do you make of a government official like Claire McCaskill?

http://www.auditor.mo.gov/aboutus/claire.htm

averros • October 10, 2006 2:27 AM

In other words, given your blanket
generalization, what do you make of a
government official like Claire McCaskill?

Her salary is paid from the money coming from my (and your) taxes. I did not consent to purchase her services, and pay for them only because otherwise I’ll be imprisoned, beaten (if I refuse to go to prison “voluntarily”) or killed (if I’ll be so incondiderate so as to resist the beating).

This means that she’s nothing more than an accomplice in a theft, a recipient of stolen goods. (If fact, she was a prosecutor, which makes her one of the thugs). The fact that she and any number of others think she’s doing “the people” (including me) a service is totally irrelevant. Her delusions do not change the fact that she (and her accomplices) make living by extortion. At least in respect to myself.

Now, what exactly I’m supposed to make of a thief?

Realist • October 10, 2006 11:28 AM

Why do you need special software to monitor opinions from other countries’ media sources? Isn’t this what intelligence analysts do already? Along with media clipping services?

Looks lke just another “jump on the DHS money wagon” scheme to me….

Davi Ottenheimer • October 11, 2006 1:54 AM

“Her delusions do not change the fact that she (and her accomplices) make living by extortion.”

I’m guessing you did not read the linked article as your response appears to be little more than a canned diatribe.

Anyway, I thought you would at least appreciate the fact that McCaskill has successfully reduced abuse of the elderly, helped protect children’s health, and (particularly relevant to your interests, perhaps) fought against illegal and excessive taxation, all apparently with no benefit to herself.

(see link above)

“After taking office she came down hard on local governments that set tax rates above legal limits, a practice many had engaged in for more than a decade. Her efforts helped convince lawmakers to allow the state to take legal action on behalf of taxpayers.”

So, contrary to your extreme and unsubstantiated assumptions, she appears to be doing exactly what you would want — catching the thieves who you might argue have taken unfair advantage of you.

averros • October 11, 2006 3:10 AM

Davi – eh?

Did I ask her to do that on my behalf?

Did I permit her to use my money?

No.

Taking someone’s property for whatever ends without permission of its owner is theft.

The fact that you may think that your money is well-spent on her salary (and so she does not appear to be a thief to you) does not change the plain truth that she stole (and keeps stealing) from me .

Case closed.

After all, when a regular garden-variety thief claims that he really needed the stuff he stolen to help the humankind – any sane person would just laugh. But when a well-groomed politico does exactly the same a lot of people think she’s good and dandy.

You may want to start using logic to check your beliefs. From the fact that you believe that what she does is good and moral immediately follows that you believe that violence towards people who did not cause any harm to anybody else can be good and moral.

Josua • October 11, 2006 10:03 AM

averros, I understand why you feel tax is theft, as I am in the same boat. The government takes my hard-earned money and, instead of spending it for the things I would have chosen, spends it for the exact opposite, stuffs it into the pockets of corrupt morons, or burns it right away for useless shit.

But let’s refine your model in one point. Any individual employee or contractor of the state does not directly steal from you, but works for money that the IRS stole from you. They get paid with stolen goods. Now you could say they shouldn’t work for stolen goods. But then again, most of us eat animals that we don’t kill ourselves.

Josua • October 11, 2006 10:38 AM

I am absolutely sure the people behind this (the department, the president and his henchmen) want this software to watch, muzzle, punish, or eliminate individual critics and opponents.

If all they wanted was an idea how the international press and the majority of their readers perceive their actions, they could easily have someone read a small representative selection of print and online media including reader comments. Any significant idea or mood that concerns the masses will turn up there.

The only logical reason to evaluate the opinion of every single publication is to identify those with certain opinions.
The only logical reason to do so, in turn, is to identify every individual author and, which works especially well with web pages, every reader too.
And the only logical reason to do so is to somehow go after every such author and, probably, reader.
The only logical reasons to do so, finally, are to watch, muzzle (deter, censor, imprison or kill), punish (imprison, torture, or kill) or eliminate (imprison or kill) every such author and, probably, reader.

Scares me down to the bone.
Yes, we all DO have something to hide: our opinions.

Davi Ottenheimer • October 12, 2006 2:31 AM

@ averros

Sorry, you completely lost me. Your description of a thief seems to be anyone who takes money and tries to use it without any oversight or responsibility. And that definition is awfully close to what you are demanding as your own right — you should be able to take money (which you would of course call “earned”) and do as you see fit with it. Am I missing something or are you in a circle of self-condemnation?

For what it’s worth, I am reminded of that famous letter by President Eisenhower:

http://www.eisenhowermemorial.org/presidential-papers/first-term/documents/1147.cfm

“it is quite clear that the Federal government cannot avoid or escape responsibilities which the mass of the people firmly believe should be undertaken by it. The political processes of our country are such that if a rule of reason is not applied in this effort, we will lose everything–even to a possible and drastic change in the Constitution. This is what I mean by my constant insistence upon ‘moderation’ in government. Should any political party attempt to abolish social security, unemployment insurance, and eliminate labor laws and farm programs, you would not hear of that party again in our political history. There is a tiny splinter group, of course, that believes you can do these things. Among them are H. L. Hunt (you possibly know his background), a few other Texas oil millionaires, and an occasional politician or business man from other areas. Their number is negligible and they are stupid.”

Josua • October 12, 2006 9:04 AM

“Sorry, you completely lost me. Your description of a thief seems to be anyone who takes money and tries to use it without any oversight or responsibility. And that definition is awfully close to what you are demanding as your own right — you should be able to take money (which you would of course call “earned”) and do as you see fit with it. Am I missing something or are you in a circle of self-condemnation?”

Davi, your analogy is false. Averros only spends money that he worked for and earned himself. Both he and his employer (or clients) deliberately choose this relationship because they both benefit from it. If as an employer or client doesn’t want to give Averros his money, he doesn’t need to do so.

But can we choose NOT to give our money to the IRS and NOT have it spent for new high-tech weapons, mass surveillance, large corporations etc, but rather for our food, houses, health, education and protection of environment? No we can’t, because the state would arrest us and take our money away with force. Yes that makes a difference.

Despite the “tax is theft” talk, I – and probably averros too – know very well that SOME expenses of the state are necessary, and cannot be funded other than with tax.

This includes at least sane democratic legislative, non-political police and jurisdiction to uphold human rights and naturally felt justice; a minimal social safety net, emergency response and probably many more, including defense to the extent that REAL threats exist and cannot be mitigated with sane diplomacy.

But politics today are a battlefield of economic interests, and the government deprobated to an instrument of the most mighty and greedy to enforce theirs. It takes as much of our money as possible without rebellion, and more (new debts) and, directly or indirectly, gives much of it to large corporations and thus their owners. It tries hard to look as if it took from the rich and gave to the poor, but in reality does the opposite. Magic tricks work because the show distracts from the double bottom.

There it starts to be theft.

It would be nice to have more choices. If each member state had it’s own independent political system, election, and tax rates, people could move to where they like and trust. Whole districts could change membership to another, nearby member state if more than, say, two thirds of residents vote for it.

You choose your wife, doctor, clothes, car, ensurances, employer, even the state to live in. The only things we can’t choose individually is our birth, diseases, accidents and other random events in life, and what government we must live with because armies of dumbs did or did not elect it. Competition would bring up better governments.

Now back to the point:

Mass opinion surveillance suppresses free speech, violates the right to vote secretly (because when your opinion is public knowledge anyway, you can as well vote in public), and finally eliminates all choices but one.

This is just one step before thought control !!!

Peter • October 12, 2006 9:07 AM

“it could be used to help the administration understand how we are viewed by people in other countries”
You presume this administration cares what anyone else thinks.

Davi Ottenheimer • October 12, 2006 1:08 PM

“Averros only spends money that he worked for and earned himself. Both he and his employer (or clients) deliberately choose this relationship because they both benefit from it. If as an employer or client doesn’t want to give Averros his money, he doesn’t need to do so.”

Point taken and expected. Thieves only spend money they steal, right? You call it stealing, they call it earning. The problem is you have failed to establish a key component of differentiation called values and ethics. Who sets those? Who resolves conflict? The bigger gun? The most money?

Moreover, I don’t know if you remember but the US has traditionally frowned on workers moving between jobs in times of national “need”. In WWII there were even federal campaigns to discredit this “bee-like” behavior because it was so disruptive due to inflation of wages and other complex economic and social factors. The point is, it’s easy to sit around and say “I should get to choose how I make money and use every penny how I like” but we don’t live in a vacuum and our decisions can impact others as their decisions impact us and we often can’t see the big picture from our living room window…so competition is great, but only so much as it exists within a system that is also sustainable.

And finally, an employer and employee can not just arbitrarily choose not to fulfill their obligations to one another. Otherwise it is, in fact, possible to be considered theft. You might not see it this way, but labor law as well as more common values and ethics disagree with you.

“But can we choose NOT to give our money to the IRS and NOT have it spent for new high-tech weapons, mass surveillance, large corporations etc, but rather for our food, houses, health, education and protection of environment?”

Well, when you do not like the terms of your relationship with the public sector then you should vote for a better representative. That was my point earlier about an auditor running for office who has lowered taxes. Why do you insist you have a right to withhold payment on a contract when you know very well that you can just negotiate more acceptable terms or prove within the system that you were overcharged and deserve a refund?

If you don’t like how things are run, you should call for moderate and reasoned change. This is a process that can work for companies or governments. Ike was right about this by saying extreme and destructive methods will do more harm than good on the whole, but change is still required. If you really want to live in a pure free market system, go to war-torn Iraq and live it up. The current situation was built by extreme free marketeers from Texas and oh, look how well their little experiment is working.

The arguments you make almost suggest that you see no justifiable good that comes from a democratic society. While I agree wholeheartedly there are bad actors and mistakes made, your decision to threfore disregard any benefit of governance is disturbing. A balanced and representative system of justice with cautious/considerate thought about freedom and liberty is something to strive for, not toss out like a bill we don’t feel we should have to pay.

Mike • October 15, 2006 11:29 AM

I cannot underestand the fuss over this.
The State Dept the CIA and other groups have a duty to know what is going on around the world. One of the ways they do that is to read foreign publications then write memos and summaries of what they find. This will include reading opinons type as well as factual articles. I would be angry if staff in our embassies were not reading the local newspapers.
Many people are already doing what this purported program does.
Whatever one might think about foreign and domestic policies, surely a better informed government is preferable to an uninformed one.

david • October 17, 2006 6:12 PM

How long till this is used for comments made in reaction to an article, like what I’m typing right now or Digg ratings? How often is US news reprinted overseas? Isn’t this really in the purview of the State Department not Homeland Security? News agencies generally do not publish “threats” so much as opinions and if the opinions are negative isn’t that a foreign relations issue? No good will come of it.

p-air • October 18, 2006 2:17 AM

This can all be done today so I’m not sure why it’s requiring a a consortium of major universities to undertake this exercise. There are a few start-ups that already can provide sentiment analysis about products and companies. Some of these are selling their service as buy & sell side research for hedge funds and financial institutions to get signals on problems that may be breweing for companies.

Anonymous • November 3, 2006 3:53 PM

When you act in a way that you know is morally right then you dont have to monitor what people think about you. This is just a way for the people in power to find out what they can get away with.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.