Schneier on Security

I, for one, welcome our new ethical robot warrior overlords.

Bender: "You know what always cheers me up? Laughing at other people's misfortune. Hahaha!"

=======
Bender: [while sleeping] "Kill all humans, kill all humans, must kill all hu..."

[while sleeping] "Hey, sexy mama... Wanna kill all humans?"

=======
Maybe if we programed robots to kill lawyers, we'd establish some time buffer for the preservation of the rest of humanity. Can lawyers truely be considered fully human?!? It's such a grey area. ;-)

One thought: The work is interesting, the spin might keep it from getting traction.

When you build your killbot [1], you want these levels of control because this is effectively "good target discrimination" and "minimal collateral damage", if you just do a search-and-replace of "ethics" with "rules of engagement"

Because even if you want to do the "Kill all humans" routines, you don't want to waste ammo shooting at Number 6 or a Sharon.

[1] "We can always make more killbots" -Bender

I have long believed that Azimov's Laws of Robotics are the only safe rules to apply.
If we teach 'thinking' machines to kill humans, how much time will we have left before they perceive that all humans are a problem to be eliminated?

I think we're way overdue for a similar set of rules on bombs. In time of war, bombs should somehow magically just know who's a good guy and who's a bad guy. No country should ever even consider using bombs unless there's no chance whatsoever of them injuring anyone except enemy soldiers, members of the enemy command and control hierarchy, and possibly those who voted for the party in power.

Let me know if I'm off base here...

y'know, pop-culture is full of depictions of what can go wrong when you build autonomous killing machines (from terminator to screamers to blade runner to that episode of st:tng with the holographic weapons salesman on a dead world)...

considering our persistent ability to realize the unintended consequences of our actions in other spheres, why do people keep trying to build these things?

it seems to me that even ethical autonomous killing machines are a bad idea...

Re: Seth Gordon

Please, go back to slashdot.

/me looks around for an inner-tube, or a flat of cardboard....

This is a slippery-slope if I ever saw one - disconnecting the human from the trigger is a very bad idea.

.02

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Now, if we could only get those who control the robots to follow the same ethical standards. Oh, maybe they already follow the standards they want the robots to use.

What is the point of having a war if it is going to end up technology against technology. Not that I condone killing people but how can there be a winner unless one side concedes?

great idea if only one side has it and is trying to save their own lives but if one has it, the others will shortly.

Wasn't there an older Startrek episode where two worlds fought their wars on computers and when your area was deemd hit, you went to a machine to terminate your existance?

Ethical autonomous military robots will predict the probability of victory in the face of an opposing force (of autonomous military robots?) and surrender without a single shot fired.

Pawn to e4 -- checkmate in 43 moves!

All right, so are we talking about Bolos? Or will Bolos carry smaller infantry-ish support robots?

(laughs)

Roses are Red,
Bolos are Blue...

What kills me is that all these scientists and others working on robots seem to have forgotten what happens when AI is given weapons to control.

See also: Terminator series, The Matrix series.

I don't see how a robot differs much from a land mine in the ethics department. Both are autonomous devices intended to kill people, perhaps the robot is a bit better at discerning between friend and foe but that doesn't lessen the responsibility of the people who fielded it.

Nobody better tell this ethical robot how our support for Mideast dictatorships is all about promoting democracy. Its head would explode.

http://news.bbc.co.uk/2/hi/middle_east/3248119.stm

I immediately thought of ED-209 from RoboCop: http://en.wikipedia.org/wiki/ED-209

C'mon people... movies are fun and all, but if regurgitating sci-fi movie plots is the best argument you can muster...

Let's make this simple... The "ethical" robots are only going to be as ethical as those in charge of those doing the programming. Take a look at what "those in charge" are currently doing to the constitution. Do you really want to trust the ethics of those people in a robot? I think not.

"An OCP product may not act against a senior official of OCP."

You're fired.

*BANG*

There are definitely security implications here. What happens when the coder makes a mistake in a key part of the program or hardware corruption changes the memory of this beast? Worse still - imagine the consequences of the enemy breaking into your robot army. You think a virus or trojan on your home PC is bad, imagine a Microsoft Soldier (Bob on steroids?) robot on the battlefield.

"Talk to the bomb. Teach it epistemology".
(The dead captain)
Seriously, I agree (from reading only the abstract above) that these "ethics" are more like "rules of engagment". Combine the propensity offleash and blood troops to occasionally to "bend" those rules with the litany of Electronic Voting Machine "errors" (and that is as charitably as I can put it), and you indeed approach the level of movie-plot Armageddon.

(OTOH, there are folks as we speak working to bring about Armageddon, so...)

Kees: go read "I, Robot" by Asimov. Not the movie, or the book adaption of the movie, the original collection of short stories. Asimov, after creating his 3 Laws, proceeded to write a good dozen stories about situations where the 3 Laws didn't work as expected.

Personally I think any "ethical constraint" rules hard-wired into robots will only be a good idea as long as the robots aren't capable of independent reasoning. Once they reach that stage, well... how do *you* react to someone forcing you to do things?

@bear
You need to read more science fiction -- try some of Aszimov's short stories on the subject, and then move on to the Star Trek and Star Trek;TNG series for similar episodes...

MikeA:

Isn't that "Talk to the bomb. Teach it Phenomenology"?

I think that no matter what we call it - even rules of engagement - we'll find that actually setting the rules so that WE agree with them will be the hardest part.

We can't even do that now in order to perfectly tell other people how to behave. There's always unforeseen loopholes.

So if we're to create a machine that can make ethical decisions as regards to taking a human life, will that machine be able to refuse an order to kill? Would any nation entrust the lives of its citizens to something can disobey orders?

Morality and ethics vary widely amongst human populations; we can't always decide for ourselves what is moral or ethical yet we're going to create machines that can? And we're going to assume that we can test the machine's morality thoroughly enough to trust it with the use of lethal force in less-than-benign environments?

I believe systems such as the one proposed are going to be a *very* long way off.

***

You can't program what you don't understand.

***

Didn't we already try this with mustard gas ? The friendly fire will be pretty awesome when it happens. The friendly robot detects soldiers and eliminates them. Can't think of anything fairer than that.

I miss the old BattleBots show on Comedy Central. No humans were hurt in the production of that show. Why can't we just fight by proxy? Best engineering wins.

Is it beyond hope that battle mechs might be controled by semi-benevolent AI, like Tweedledee and Tweedledum?

Tachikomas for the win, I say

If you tell a robot tank to shoot a group of prisoners, does that absolve you of responsibility for a war crime?

From the short version, it sounds as if this may be a much more sophisticated version of the limits coded into CIWS and other self-targeted weapons to stop them from shooting into or through the installation they're supposed to defend.

@possibly those who voted for the party in power.

I'm trying to figure out how that would work in Florida. Do they have to *know* that they voted for the party in power? (i.e. does it need to know their intent?)
And if they intended to vote for the party in power, but failed due to hanging chad, are they spared?

If an autonomous military robot violates the Geneva Conventions, would its programmer be liable for war crimes?

For those who wouldn't recognize Ronald Arkin's name, he literally wrote the book on modern robotics; see "Behavior-Based Robotics."

The article does not mention Terminator style movies but he does directly address Asimov's laws:
----------------------------------------------------------------

I suppose a discussion of the ethical behavior of robots would be incomplete without some reference to [Asimov 50]’s "Three Laws of Robotics" (there are actually four [Asimov 85]). Needless to say, I am not alone in my belief that, while they are elegant in their simplicity and have served a useful fictional purpose by bringing to light a whole range of issues surrounding robot ethics and rights, they are at best a strawman to bootstrap the ethical debate and as such serve no useful practical purpose beyond their fictional roots. [AndersonS 07], from a philosophical perspective, similarly rejects them, arguing: "Asimov’s ‘Three Laws of Robotics’ are an unsatisfactory basis for Machine Ethics, regardless of the status of the machine". With all due respect, I must concur.

The long-standing Sci-Fi prophecy of intelligent machines rising up to enslave and destroy the human race has disappeared from modern culture.

As far as I can tell, this coincided with the release of MS-DOS.

When my company made plans to work in the defense and security sector, I was tasked to come up with a set of rules defining how far we would go there.

For me it is generally immoral to create or build a machine that automatically kills people. The mentioned Land Mines also fall into this category, as well as systems like the Oerlikon GDF-005 robot cannon that killed nine soldiers in south africa in October 2007.

When I build a gun and someone pulls the trigger, he has to cope with it.

When I create a rule based system that pulls the trigger, it is actually me who makes the final decision who gets killed.

Even if I could create a perfect and flawless system, it still might kill anyone under certain circumstances, and that includes myself or my kids.

However, in reality every such system will be heavily flawed, and it *will* kill innocent people sooner or later, even if it is a simple system like a land mine.

Using a complex AI-based system today or even in the foreseeable future for the purpose of making kill decisions is stupid and dangerous bullshit, given that we did not make much real progress in the last 60 years of AI.

I think *any* machine with the purpose of harming people that does not require a conscious and present human decision to do so should be outlawed everywhere and forever, no exceptions.

Autonomous killing machines are not only a very bad idea, they are highly condemnable.

No amount of built-in ethics can change that.

If we are concerned about a future desensitized to war, we're already there. I feel so far removed from the daily horrors of a war my country is currently fighting that it is almost like it is already being fought with robots rather than humans.

http://www.samsungtechwin.com/product/features/dep/SSsystem_e/SSsystem.html

All Your Base Are Belong To Us!

It all sounds pretty scary, and really, really sad.

When machines fight against humans, humans are always the good guys

"The creation of techniques to permit the adaptation of an ethical constraint set and underlying behavioral control parameters that will ensure moral performance, should those norms be violated in any way, involving reflective and affective processing."

That is far and away the scariest thing I have ever read in the context of weapons development.

The ability to detect and select new targets effectively on the battlefield would be very cool, the ability to analyze and infer new rules about target selection, based on machine interpreted rules about morality, not so much.

Kyle has nailed this, especially when looked in the light of asymmetric warfare; using robots to do the dirty work of war will only enhance the morale of those being shot at. Think of it from the other side: "they're so cowardly that they have to send machines to kill us."

The slippery slope began with self-propelled torpedoes and naval mines. Land mines followed soon enough.

Now we will see autonomous weapons actively discriminating among friendlies, hostiles, and neutrals. In the 'weapons tight' condition the rule is to kill only hostiles; in 'weapons free', it will kill anyone not friendly.

What exactly does a hostile look like? A friendly? A neutral What algorithm will tell them apart?

If we equip friendlies with transponders, then most of the time the automaton won't kill them, but it will kill all the non-transponders it can -- hostiles, neutrals, and unlucky friendlies. Building better transponders might seem like a good idea, but the money is made mass-producing them, so once mass-production begins, mass-counterfeiting begins.

Killbots would be safer than indiscriminate weapons like a hydrogen bomb, but why would we want a weapon that will kill our own kind, and will kill neutrals indiscriminately? I thought the whole point of war was to make fortunes building weaponry, and then make bigger fortunes replacing the weaponry destroyed or outmoded.

Then there is the fundamental problem sketched by Korzybski; semantics. Excuse my fumbling to bring this to the surface. Arkin may be wise; but words are abstractions linked to real-world processes by intellectual structures - which humans construct and use differently.

Whose "ethical" are we using?

The complaint about dredging up Sci-Fi scenarios is that Sci-Fi and other speculative fiction tends to be better at illuminating dystopias rather than utopias... because dystopias tend to be what we NEED to worry about.

Beyond that, looking at what passes for political leadership we have, who set the ethical rules? Who reviews them?

Personally, I want a living thing somewhere in the decision loop.

I look forward to these rules as an addition to Genevia Convention rules - (or a new version of them). Understanding how and when robots should be used in warfare is something we should do.

I don't think it would be too hard for one disciplined in the arts to take an RC car, add an imaging sensor (for face/place recognition), and program it to carry a "package". I would prefer there be rules about how close the next non-target person is for a device like that. (Of course - I would prefer this thing to not exist at all...)

Granted - not all people would agree or use these rules. But the more countries that do, the safer each of us is going to a baseball game.

This country is already wiping its ass on the Geneva Conventions.

As a think piece, it's barely O.K.
As a command rubric, the bias
in favor of presuming total obedience of a system is a serious - perhaps literally fatal - flaw.

Politicially correct autonomous killing machines? Shirley, you jest...

We don't need fancy ST plots with simulated war, or even fanciful EMP devices like The Matrix.

All combatants just need to wear CAPTCHAs to confound the machines for days.

@ all those who called this reprehensible
Amen! War is bloody and -usually- profitable. Without the blood, it's just profitable. War without blood = murder for hire.

The robots will become an extension of our actions, both in our factories and in our armed gestures. Is it desirable to choose a model of human cognition to draw up behaviour (with its invariable weaknesses and the ability to use social engineering to neutralize them) to be this model? The era of Terminators seems close if one considers that modern homes will soon present several robot like the VCR and it was less than twenty years ago.

We're trying for the wrong standard. Trying to make 'ethical autonomous weapons' is not the issue, trying to make 'more-ethical-than-humans autonomous weapons' is undeniably an excellent idea. While it may well be very difficult to make truly ethical robots, the standard of making robots that behave themselves better than humans is pretty easy, really.
It should be noted that armies go to considerable efforts to prevent their soldiers making decisions, hence 'rules of engagement' rather than an instruction to 'be ethical'.
We worry about robots turning against us and destroying us, but we should remember that humans have been turning against each other and destroying each other for a long time, and it's still easier to make more people than it is to make more robots. When robots have a mechanism in place to evolve, then we're in trouble.

This kind of work also reminds me a bit of the work Mark Stefik at PARC did in the 90s on rights definition languages for digital content -- machine-readable, formal languages that would allow content publishes and content purchasers to negotiate how much the purchasers would pay for how many copies made under what conditions and so forth. It was very elegant, but since the only position most publishers wanted was "We keep everything. Take it or leave it." the idea never reached terribly widespread distribution.

Didn't Norbert Wiener (of "Cybernetics" fame) discuss this (the ethics/morals of automated systems that kill people) in some of his works 50 or more years ago? I can't think of the titles off hand. "God and Golem" or "The human use of human beings" might be the ones.

It comes down to a very complex issue of where do you draw the lines. How is sending targeting information to a B52 (with a human pilot) flying above the clouds different than sending the same targeting info to a ICBM?

It is not clear if this type of ethical control and reasoning system for robots could prevent incidents like: a) the genocide in Rwanda, b) ethnic cleansing in Kosovo, c) forcing children to fight in armies like in Liberia and Congo, d) cutting off a hand of the enemy to overload their medical facilities as in Liberia. Once these robotic war fighters are developed they would become the prime objects for sale by unethical(?) arms dealers. What's to prevent their algorithms from being altered to gain an advantage in combat. Will the software designers who made a mistake or an unauthorized alteration be charged with a war crime?

Oh, how cute. The people manifestly lacking any ethics (in that they think nothing of wasting money expropriated from people on advising how to do automated mass-murder "ethically") are writing about robot ethics.

There's nothing complex about the issue of killer robots. They're simply elaborate traps - and all "ethics" of such traps comes from the intent of their use by their owners.

Bruce, you missed another "daunting problem": doubt.

Even if we nail the problems you list, we still end up with a bunch of killing machines that are absolutely certain that they're doing the right thing. History shows that that's a recipe for genocide.

@Jack C Lipton

"Personally, I want a living thing somewhere in the decision loop."

What if that living thing were:
* Dick (shoot-for-the-face) Cheney?
* Charles Manson?
* Ted Kazinski?
* Karl Rove? (generic neocon example)
* George Bush? (generic Forest Gump example)

These implications were all worked out in science fiction in the fifties and sixties. The first (I think) was "Cordwainer Smith's" manshonyaggers (menschenjaggers, or manhunters), which were still patrolling their area after 50,000 years. Probably the most thorough were the Berserker stories of Fred Saberhagen. I remember the BOLOs, too. Norman Spinrad wrote the Star Trek story "The Doomsday Machine," which introduced the idea to a wider audience without any real ethical implications. There must be hundreds of stories laying about in moldering stacks of old Galaxy, Analog, and Amazing magazines.

It takes twenty years to raise a soldier, and months to train him well.

If you can manufacture and program them by the millions, it's much cheaper in terms of time, emotion, and supply. Attrition is now an attractive option. Think of fighting the Iranian or Chinese "human wave" strategy with a "robot wave."

However, the U.S. combined arms strategy is far more effective in terms of massed battle. Bombs are much cheaper than robots

But think of sending a robot with advanced AI and a machine gun into a crack house in Detroit or a safe house in Baghdad. The thinking is to save some good guy lives.

But it will be just as safe and still more effective to send in a robot controlled by a human hand and mind, such as the flying drones currently employed over Iraq, Afghanistan, and Israel.

For the forseeable future, robot warfare will probably be defensive, i.e., a machine gun that fires on anything that crosses an infra red beam. Kind of like an automated grocery store entrance, only noisier.

@aikimark

Well, think of the two-man rule for launching nuclear weapons as part of the fail-safe, but, yeah...

"I built the M-5 with my engrams" - Dick Cheney.

If they built the M-5 with Dubya's engrams it wouldn't be able to add, much less do any real damage.

@aikimark

Well, think of the two-man rule for launching nuclear weapons as part of the fail-safe, but, yeah...

"I built the M-5 with my engrams" - Dick Cheney.

If they built the M-5 with Dubya's engrams it wouldn't be able to add, much less do any real damage.

The whole thing seems like a massive exercise in question-begging to me.

"If only we knew how to make an autonomous battlefield robot, we could make an autonomous ethical battlefield robot, if only we knew how to make it ethical".

Things don't always go according to plan when you allow a device with limited capacity for reasoning onto the battlefield.

It brings to mind the Soviet "Anti-tank Dogs":
http://en.wikipedia.org/wiki/Anti-tank_dog

A great deal of faith is being placed here on the idea that the generals, civilian policymakers, and their minions in the R&D industries want the troops to behave well, and it's just the bad apples who muck things up. A lot of history suggests otherwise. Maybe we need robot robot-programmers, as well...

In contrast to most people's comments, I think such an ethical
robot could be a very good thing, even considering myself a pacifist.

As history has shown, war is inevitable. Most unfortunate.
As history has shown, soldiers commmit crimes and kill innocent people.
Most unfortunate, too.
As history has shown, software has bugs, so will the robots,
and kill innocent people. Most unfortunate, once more.

The question is: who is more likely to follow ethical rules and the
Geneva convention in extreme situations?
Who will end up with the smaller number of killed innocents,
a group of n soldiers or n robots?

Prof Noel Sharkey to address this at University of Sheffield today:
http://www.physorg.com/news123306321.html