Detecting Betrayal in Diplomacy Games

Interesting research detecting betrayal in the game of Diplomacy by analyzing interplayer messages.

One harbinger was a shift in politeness. Players who were excessively polite in general were more likely to betray, and people who were suddenly more polite were more likely to become victims of betrayal, study coauthor and Cornell graduate student Vlad Niculae reported July 29 at the Annual Meeting of the Association for Computational Linguistics in Beijing. Consider this exchange from one round:

Germany: Can I suggest you move your armies east and then I will support you? Then next year you move [there] and dismantle Turkey. I will deal with England and France, you take out Italy.

Austria: Sounds like a perfect plan! Happy to follow through. And—thank you Bruder!

Austria’s next move was invading German territory. Bam! Betrayal.

An increase planning-related language by the soon-to-be victim also indicated impending betrayal, a signal that emerges a few rounds before the treachery ensues. And correspondence of soon-to-be betrayers had an uptick in positive sentiment in the lead-up to their breach.

Working from these linguistic cues, a computer program could peg future betrayal 57 percent of the time. That might not sound like much, but it was better than the accuracy of the human players, who never saw it coming. And remember that by definition, a betrayer conceals the intention to betray; the breach is unexpected (that whole trust thing). Given that inherent deceit, 57 percent isn’t so bad.

Back when I was in high school, I briefly published a postal Diplomacy zine.

Academic paper.

Tags: academic papers, cheating, games

Posted on August 10, 2015 at 3:32 PM • 25 Comments

Comments

Smirk • August 10, 2015 3:49 PM

Maybe this researcher should also work on a program to detect corruption as it woulndt be very diffrent, or am i wrong?

Matt • August 10, 2015 4:00 PM

57% may be way better than human accuracy, but it’s not a particularly useful thing to rely on. Consider that if you are betrayed on (say) 1 out of 10 turns, and you use this computer algorithm to predict betrayal, the computer will predict that you’ll be betrayed 4 times during those 10 turns. That’s a lot of false positives, and if you act on them, you’ll end up betraying people who had no intention of betraying you (or at least, not on that turn). Not to mention that about half the time that you ARE going to be betrayed, the computer won’t predict it accurately.

Even if the computer was 90% accurate, that would mean you’d still have nearly one false positive for every time you actually get betrayed.

tyr • August 10, 2015 4:02 PM

I have never been in a Diplomacy game where the
players were playing the current game, they were
all playing for revenge from past games. It is
deceptively simple but can bring out the best
or worst in human character.

We actually won one game without playing by
saying the traditional alliance to the other
players. They forced a redraw of nation states
and we declared it the easiest victory ever.

Social engineering at its best.

Mike Barno • August 10, 2015 5:30 PM

[Longtime lurker, first time poster.]

Like Bruce, I am a former publisher of a play-by-mail Diplomacy zine. (His was 1980-81, mine was 1979-83.) Over decades I did amateur study of styles and content and linguistics of Dip negotiations, at one point going so far as to wear pink bunny ears during a tournament and noting its effects on other players.

I never found any particular word choices, syntax, or other algorithm-measurable factors that had a consistent factor recognizable as “about to stab”, nor as the less-definable “about to be stabbed”. The latter sounds to me like a back-formation from looking for patterns in data regardless of possible causation.

Far more predictable a “tell” is, as the linked article noted, a change in attitude. Suddenly more polite? Suddenly gruff? Likely either planning a stab, or paranoid that you’re planning to stab him. And, of course, all the classic poker tells apply in face-to-face play: tics, inability to look you in the eye, frog in throat, jumpiness.

trhaffey • August 10, 2015 6:51 PM

The best way to know when someone is going to sstab you or brtray you is to know the person. after you play with people in many games you will learn how they act when they are setting you up or are going to abandon an alliance. You can just tell, Like the study showed, they become less responsive to emails, vauge or misleading in their responses or stop sending emails all together. A change in attitude I guess. But, playing with strangers is a different thing all together. When I am playing strangers I always let them open negotiations and see what they are saying. If they never say anything you can figure they are pretty new to the game and you will want to attack them as soon as possible no matter what they say. Newbies tend to be dot grabbers and you can get them to do things that will make the other players mad at them. Newbies tend to leave the game first. Eenough of my midday ramblings.

Slime Mold with Mustard • August 10, 2015 6:56 PM

I played Diplomacy when it was still an Avalon Hill board game, and I yet a lad. It (in part) taught me the cynical attitudes I retain: Sudden and unwarranted friendliness often foretells attack or other exploitation.

“More work is needed to explore whether these patterns exist in real life”. Uhm, YES they do!

Jimmy Hoffa would have fared better to understand this.

Mike Barno • August 10, 2015 8:30 PM

From a footnote in the study’s PDF, section 4:
“In rare cases, the betrayal can be mutual (i.e., both play-
ers start attacking each other in the same season). In such
cases, we consider both betrayals.”

In reality, “mutual betrayal” is very common, especially in tournament play. Most players have some instinctive sense, not considered as linguistic algorithms nor as percentages of correlation, but just as feeling hinky when an ally seems less natural. In addition, most good players can read a board position and see that it’s ripe for a stab by a Great Power who can get away with it. So both parties often end up disconcerting each other into reactions whether they fully intended to fight or not.

Also, I would have more respect for the drearily academic study’s paper if the authors could at least write a support order correctly. From a footnote in the PDF, section 2:
“For example, if an English army
in Belgium is attacking a Germany Army in Ruhr, a French
army in Burgundy could strengthen that attack. This is ac-
complished by the French player submitting a move explicitly
stating “I support England’s attack from Belgium to Ruhr”.”
The correct order by the 1971 or 1976 Avalon Hill rulebooks, or pretty much any Dip rules except the 1961 Games Research Inc original, would be: “A Burgundy S ENGLISH A Belgium-Ruhr”. (The nationality for a foreign support is optional, but it’s helpful, so I always added it in adjudications when I gamemastered.)

cynical • August 10, 2015 8:50 PM

Thus anyone selling trusted root is setting you up for one.

Mike Barno • August 10, 2015 8:53 PM

@ trhaffey:

“If they never say anything you can figure they are pretty new to the game and you will want to attack them as soon as possible no matter what they say. Newbies tend to be dot grabbers and you can get them to do things that will make the other players mad at them. Newbies tend to leave the game first.”

No. Two main points:
(1) If everybody treats the newbies that way, then most of them never return. The group doesn’t grow and get more diverse thinking; instead it shrinks to a self-reinforcing clique of Dip stereotypes that fades away when people quit. I’ve seen it happen more than once in local clubs and in postal zines; I’m sure it happens more online, which was the focus of the study.
(2) A better player [ask multi-time North American champ Chris Martin for example; he provided datasets for the study] knows there are advantages in helping a newbie as an ally, giving him good advice, supporting him forward, and swooping in to stab much later, when it gives him his 17th and 18th supply centers, thus the victory.

Mike Barno • August 10, 2015 9:18 PM

So I am still less impressed after reading the “Relevance Beyond the Game” and “Conclusions” sections. The best they can point at for relevance is online romance. The conclusions are, roughly, “We didn’t expect to find much ability to detect betrayal, we pretty much didn’t, but we find new insights” (without summarizing any). And “we could extend this methodology to Wikipedia edit wars.” Nothing about information security: In face-to-face there’s leaning over somebody’s shoulder as she writes, or allies showing each other their orders, or someone grabbing a suspected stabber’s orders out of the box before turning in his own. In PBEM or Judge-website play, you get all the traditional online security issues that this blog regularly discusses in other contexts. Whether FTF or online, all those InfoSec topics are primarily about detecting betrayal. Not touched on in the study.

And the full list of references has nothing Diplomacy-related except the Richard Sharp book. It feels like a couple of college students discovered a Dip website, had fun for a couple of months, and said “We can write our term paper about this, it’s easier than doing real research and real work!” The linguistics was the only part of it that fit the methodology required for a course.

a ghost only you can see • August 11, 2015 2:06 AM

Plan revealing probably can make someone a victim… because you are revealing your plans. 🙂 Lol.

Politeness, if insincere, can be an indicator. But it can also be someone who is normally polite and simply holding their tongue.

There are a number of studies involving betrayal, over the years.

In a Judas situation, it tends to be more about the many cues people put out showing their belief in the group message, or lack thereof. People take great pains to communicate their level of engagement, and it can be very difficult to fake engagement. On a professional level, I think that is where there is a strong inconsistency: very profound signs of very strong engagement, and inconsistent signs of zero engagement at all. Over doing signs of engagement is a major indicator.

We tend to miss these cues only because in normal behavior, it is routine to feel a need to fake engagement to get along.

And, because we generally do not stop and think, “Why is this buzzer in my gut going off”, to attempt to bring to conscious evidence what one’s unconscious is picking up on.

You have to really put a lot of work into being able to get that level of inner observation communication between your unconscious and your conscious.

This may include that ‘if someone is not who they present themselves as’, then who, exactly are they really.

Expectation there helps a lot, though can deliver a lot of false positives. If you are not considering it remotely conceivable someone is not as they present themselves as, and have no reason to think this, you will be less likely to see it.

This can operate post-mortem, too. In retrospect, people can look back and figure out all the cues they missed before, after they discover someone ‘probably was actually not who they thought they were’.

Evan • August 11, 2015 4:48 AM

I buy this. A shift into a more polite register can indicate that the one party recognizes their vulnerability to betrayal, and attempts to use normal social techniques to ingratiate themselves to the person to whom they are vulnerable as a means of mitigating that risk. If Austria is in a position to betray Germany, making Austria more well-disposed to Germany to prevent or forestall betrayal is a coherent strategy. It’s principally the same technique when you make small talk with someone before asking them for a favor.

Unfortunately, Diplomacy players are a ruthless lot and it doesn’t seem to work very often.

Sam • August 11, 2015 6:17 AM

it’s not a particularly useful thing to rely on

I don’t think it’s about tips to help you “win”, the lesson here is about how people assess risk – in this case risk of betrayal within the scope of a board game.

Clive Robinson • August 11, 2015 6:53 AM

@ Sam,

… the lesson here is about how people assess risk – in this case risk of betrayal…

And back in the real world that led initialy to the Korean War (Stalin dumped it on china), the Vietnam War, Mutually Assured Destruction, God alone knows what in South America and the Middle East, and eventually the invasion of Iraq… the game carries on with Inter State Dispute Resolution in all current US trade agrement negotiations effectivly “at the point of a knife to the throat”…

They say art mimics life, it might just be both, especially when you know the board game was a favourit of Nixon’s advisors and cronies, reputedly more so than poker.

Brian • August 11, 2015 9:37 AM

Skims paper. Apparently, 57% is better than chance (52%). Two concerns I have with this study’s conclusions:
1) Even trusting that their stastics promise the 57% has a high likelyhood not to be due to chance, it could still be caused by somehow allowing their training data to pollute their test data.
2) Politeness can be correlated with betrayal, rather than being an indicator of betrayal. An alternative explanation is that the current game state causes a combination of politeness and betrayal. This is somewhat supported by the paper; the victim’s behavior also changes immediately before a betrayal.

Ray Dillinger • August 11, 2015 9:40 AM

Odd. They don’t find a huge amount of statistical correlation (%57) but this sounds to me like a pretty accurate description of office politics at a few places. Um, possibly places where I’ve worked. In the past. You know.

Terry • August 11, 2015 9:55 AM

Diplomacy is an under appreciated game. In the early seventies in a small college town I played Diplomacy against (among others) the leader of the local weather underground branch of SDS. He insisted that we change the rules to allow written (and therefore enforceable) treaties. I argued against this for quite a while as it defeats the essence of the game, but finally agreed. The game began, everyone except for the weather underground leader signed a written treaty, we all attacked him on the first turn and he was wiped out. QED.

albert • August 11, 2015 10:25 AM

What are these folks gonna do when they get out of school? I hope they find gainful employment. We’ve got too much drag on the economy already. Is there a golden toilet seat award for academic papers?
. .. . .. o

Sam • August 11, 2015 11:21 AM

@Clive Robinson

the Korean War (Stalin dumped it on china), the Vietnam War, Mutually Assured Destruction, God alone knows what in South America and the Middle East, and eventually the invasion of Iraq…

Yeah, you’ll note that I put “win” in quotes, deliberately. There’s the “win” where you end up better than your opponents do, and there’s “win” where you finish better than you personally started. These can often be orthogonal; maximising one doesn’t mean positive change in the other.

In any case, I don’t think anyone would suggest attempting to reduce any of those conflicts to a deterministic decision process either. Well… probably some people would, and maybe those people are the ones we should try to keep out of office.

Clive Robinson • August 11, 2015 12:43 PM

@ Sam,

Well… probably some people would, and maybe those people are the ones we should try to keep out of office.

If only we could…

I think it was Douglas Adams, who had the throw away line, about a planet of humanoids who kept voting for a lizard as president, and when asked why they kept voting for lizards replied “They had to, because they did not want the wrong lizard getting into power”….

Sam • August 11, 2015 3:38 PM

Yeah, it was Douglas Adams, it was from Hitchhiker – Ford is explaining some situation and Arthur gets to be bewildered. There was a bit of setup, acknowledging that the lizards and humans hated each other and everyone knew it.

It’s one of my favourite Douglas Adams quotes – I didn’t have to look up the quote but I did have to check how many h’s go into Hitchhiker.

tyr • August 11, 2015 8:50 PM

My favourite DA moment was the planet wiped\
out by plague because there were no telephone
wipers anymore.

Most Abo descendants love written treaties
because you can see what happened.

Clive Robinson • August 12, 2015 4:03 AM

@ Sam,

I didn’t have to look up the quote but I did have to check how many h’s go into Hitchhiker.

🙂 as Douglas once said, “one word or two” and “to hyphen or not to hyphen that is the question”. Likewise I’m not sure anybody realy knew how it was spelled at any point in time.

Mind you our host has occasionaly tried out “gazzilion” and even Google had trouble with that, and my pet peeve is the spelling and meaning of “in-it” as heard in parts of London “beyond the Half tide lock”.

Mike Barno • August 12, 2015 12:59 PM

Hey, here’s an easy one to add to this algorithm for its next test:

If your ally sends a sedanful of intimidating men to your wife’s house, and they tell her they know she’s married to the player of $GREAT_POWER in $DIPLOMACY_GAME, and they try to get information from her…

Then your ally is probably about to stab you.

Jeremy • August 18, 2015 1:17 PM

“An increase planning-related language by the soon-to-be victim also indicated impending betrayal”

My first thought here is that increased planning and increased chance of betrayal are probably both correlated to the volatility of the board position.

Schneier on Security

Detecting Betrayal in Diplomacy Games

Comments

Leave a comment Cancel reply