Fraud Detection in Pokémon Go

I play Pokémon Go. (There, I've admitted it.) One of the interesting aspects of the game I've been watching is how the game's publisher, Niantic, deals with cheaters.

There are three basic types of cheating in Pokémon Go. The first is botting, where a computer plays the game instead of a person. The second is spoofing, which is faking GPS to convince the game that you're somewhere you're not. These two cheats are often used together -- and you see the results in the many high-level accounts for sale on the Internet. The third type of cheating is the use of third-party apps like trackers to get extra information about the game.

None of this would matter if everyone played independently. The only reason any player cares about whether other players are cheating is that there is a group aspect of the game: gym battling. Everyone's enjoyment of that part of the game is affected by cheaters who can pretend to be where they're not, especially if they have lots of powerful Pokémon that they collected effortlessly.

Niantic has been trying to deal with this problem since the game debuted, mostly by banning accounts when it detects cheating. Its initial strategy was basic -- algorithmically detecting impossibly fast travel between physical locations or super-human amounts of playing, and then banning those accounts -- with limited success. The limiting factor in all of this is false positives. While Niantic wants to stop cheating, it doesn't want to block or limit any legitimate players. This makes it a very difficult problem, and contributes to the balance in the attacker/defender arms race.

Recently, Niantic implemented two new anti-cheating measures. The first is machine learning to detect cheaters. About this, we know little. The second is to limit the functionality of cheating accounts rather than ban them outright, making it harder for cheaters to know when they've been discovered.

"This is may very well be the beginning of Niantic's machine learning approach to active bot countering," user Dronpes writes on The Silph Road subreddit. "If the parameters for a shadowban are constantly adjusted server-side, as they can now easily be, then Niantic's machine learning engineers can train their detection (classification) algorithms in ever-improving, ever more aggressive ways, and botters will constantly be forced to re-evaluate what factors may be triggering the detection."

One of the expected future features in the game is trading. Creating a market for rare or powerful Pokémon would add a huge additional financial incentive to cheat. Unless Niantic can effectively prevent botting and spoofing, it's unlikely to implement that feature.

Cheating detection in virtual reality games is going to be a constant problem as these games become more popular, especially if there are ways to monetize the results of cheating. This means that cheater detection will continue to be a critical component of these games' success. Anything Niantic learns in Pokémon Go will be useful in whatever games come next.

Mystic, level 39 -- if you must know.

And, yes, I know the game tracks works by tracking your location. I'm all right with that. As I repeatedly say, Internet privacy is all about trade-offs.

Posted on November 3, 2017 at 6:35 AM • 33 Comments

Comments

willmoreNovember 3, 2017 7:01 AM

As a player of an older Niantic game, I've been watching their responses to cheating for quite a while.

The group aspect, as you mention, is where it really becomes interesting. And it only explains part of the motivation to cheat.

Generating higher level accounts and gear (Pokemon in the case of PoGo) has two uses. The first is to generate a strong account that can be used for a disposable purpose--destroying a high level target in an obviously bogus way--or for comercial sale (and in this case gear as well is a commodity).

The latter can be addressed by preventing the goods from changing hands or destroying them once they do. For gear, 'changing hands' is easy to detect. For transfer of accounts, it's much harder. But, if the fraudlent accounts can be detected than imposing limits on them--even if there is an element of randomness to it--can effectively devalue the product enough to deflate the market for the product.

The former use of fradulent accounts is the more damaging in terms of game play. All that a spoofer/botter needs to do is to grind an account up to level 8 and then deploy it to a remote location to take out a strategic target. Once the target is destroyed, there is very little Niantic has shown to be willing to do. Given the interconnected nature of events, it is very difficult to just 'roll back the database' to before when the spoofed attack happened. Events have been lost because of this use of spoofing/botting. And it has proven to be a huge impact on player morale.

I don't know if any degree of AI will help in detering these behaviors. A human and a handful of heuristics can certainly do a good job of it. But the problem is that knowledgable humans are in low supply and companies like Niantic have a low incentive to combat these behaviors.

Jenny JunoNovember 3, 2017 7:39 AM

Shadowbans are one of those things invented by engineers who haven't paused to consider the human aspect of false positives. Basically its a way to prevent the person on the business end of the shadowban from reacting to the ban. That's great if the ban is legitimate. But when the ban is the result of a false positive the punishment itself includes reduced accountability for the accuser.

I believe Bruce has on occasion referred to the War on Terror as a war on dignity because of the way people falsely caught up in it are subjected to dehumanizing treatment with very little recourse. Shadowbans are an example of that same problem - simply being suspected of being a bot results in being treated like you are not a person.

From a technical perspective its also flawed as that actual bots can take countermeasures to detect being shadowbanned - since they are automated, they can do things like statistical analysis to see if the game is treating them differently. But regular people will not have those tools available to them. So at best its just an escalation in the arms race while the actual people mistakenly caught up in it end up as the causalities.

LukeNovember 3, 2017 8:43 AM

The proper spelling is Niantic, but you used Niantec. A bit pedantic, but in case you're looking for accuracy.

Also how did a busy man like you reach level 39?! I've been playing since the game came out and I'm level 31. Also, go mystic!

GeorgeNovember 3, 2017 9:03 AM

Cheaters are going to cheat--so what stops cheating in the real world? Is there any recourse that non-cheating players have to report cheaters, or can they shun or avoid those they see as suspicious?

Bruce no doubt reached 39 due to the scope and frequency of his travels (including lots of tiny amounts of downtime). I also suspect he was an early adopter.

David SmithNovember 3, 2017 9:19 AM

You're Team Mystic? That's it, I can never respect anything you write again ;)

Niantic, the developers of Pokemon Go, first made (and still maintain) another game, Ingress. It uses a lot of the same underlying data (Ingress portals, and Pokemon Go gyms and pokestops, are often in the same places). But they use it differently -- Pokemon gyms are standalone objects, but Ingress' portals can be linked with other portals, sometimes miles away.

Niantic makes a lot more information available to Ingress players. There's a public map available to players that shows portal locations and basic information worldwide. At least for this game, making this info readily available hasn't hurt the player base. A similar "official" offering for Pokemon Go would remove the need for third-party trackers, and would potentially curb one of the major reasons bogus accounts have to be created in the first place.

Meher Baba FanNovember 3, 2017 9:23 AM

I thought this game had been banned for being a waste of vital life energy , depleting non renewable time units and crushing creativity?

WaelNovember 3, 2017 9:42 AM

@Luke, @George,

Also how did a busy man like you reach level 39?!

Three answers:

One:
Obama was a busy man and still reached level 9 on Grand Theft Auto! (first minute.)

Two:

These two cheats are often used together -- and you see the results in the many high-level accounts for sale on the Internet.

Emphasis, mine.

Three:

Bruce no doubt reached 39 due to the scope and frequency of his travels

Umm, perhaps he exchanged some frequent flier miles for level upgrades? Most likely just free upgrades: Upgrade to first class at the checkin gate + two or three levels on the game.

TatütataNovember 3, 2017 9:51 AM

I became aware of Pokémon Go cheating when I entered "GPS spoofing" in a search engine and not quuite getting the results I expected.

I ought to get one of them fondleslabs. I can feel so lonely on the bus, when I'm often the only one not using his thumbs. Even the driver quickly pulls his own one during stops.

Bruce SchneierNovember 3, 2017 9:52 AM

"Also how did a busy man like you reach level 39?! I've been playing since the game came out and I'm level 31."

I walk a lot anyway -- in the Pokemon-rich environment that is Cambridge -- and now play while walking. And the game gave me something that wasn't the election to obsess about last year.

Bruce SchneierNovember 3, 2017 9:56 AM

"Bruce no doubt reached 39 due to the scope and frequency of his travels."

Partly, but more importantly that's how I caught them all.

Actually, I caught them all exactly once for about a week: after I caught Corsola in Florida and then Farfetch'd in Seoul, and before Mewto was released in Japan. If things go well, I will again have caught them all today. I caught my first Suicine yesterday, and I will be going on my first Mewto raid in an hour.

Bruce SchneierNovember 3, 2017 9:57 AM

"You're Team Mystic? That's it, I can never respect anything you write again ;)"

What did I know? I liked the blue, and the description made the team seem science-y.

ClauClauClaudiaNovember 3, 2017 9:59 AM

Instinct level 35 here. ;-)

I'll note one more form of cheating: multi-accounting. That's more 'retail level'--I can't think of a way in which it scales to become as big an issue as the others can. But it is a factor--you log into your Valor account to knock down your Mystic gym and then recapture it as Mystic....

As far as Bruce leveling up, if you're in an urban area and are motivated, and spend a bit of actual money on lucky eggs to double your earned XP, you can level up with only short daily bursts of activity. I can't speak to his methods!

To George: Most of this cheating can be done in ways that it is hard for humans to observe. I assume there's a human report mechanism, but I wouldn't expect it to be as effective as properly tuned algorithms.

Like willmore, I played Niantic's previous game (Ingress) and have watched their (in)ability to deal with cheating with interest. I will note that Pokemon Go doesn't currently have high strategic value targets in the way that Ingress does--even a remote gym held is just one more gym; with a limit on earnings per day from defending gyms, no one gym is *that* desirable. In general, the player base benefits from churn (frequently changing gym possession). (By contrast, while Ingress players as a whole also benefit from churn, in that game one portal can be an anchor for a continent-spanning field, and records for holding a portal for a long time have in-game repercussions.)

I never saw much evidence that Ingress successfully dealt with spoofing when the in game stakes seemed much higher, so I'll be interested to see what's different now. The in game stakes seem much lower, but the commercial stakes far greater!

ChrisNovember 3, 2017 10:00 AM

David Smith: there is a booming market in cheating by extracting more information from the public map then Niantic intended. The public map does not show you every action a player takes but by monitoring the state of every portal in the game, trackers have been able to track player movements more persistently then intended. This has recently been highlighted by way of data leaks.

https://brokersguild.wordpress.com
https://plus.google.com/+Ingress/posts/KGEVPAr1JZu

Bruce SchneierNovember 3, 2017 10:00 AM

One thing I should have added: Niantic's motivations w.r.t. cheating are more complicated than simply wanting to stop it. Their primary motivation is to keep the players who are paying to play. If they are also cheating while doing so, it's in Niantic's interests to let them cheat as long as it doesn't disrupt the rest of the game.

Bruce SchneierNovember 3, 2017 10:04 AM

"I'll note one more form of cheating: multi-accounting. That's more 'retail level'--I can't think of a way in which it scales to become as big an issue as the others can. But it is a factor--you log into your Valor account to knock down your Mystic gym and then recapture it as Mystic...."

Multi-accounting was a far more useful form of cheating in the old gym meta. It made a lot of sense to have an account belonging to another team to shave a single Pokemon out of a gym and then add one back from your primary account. With the new gym meta, that kind of play makes much less sense.

I see people with multiple accounts all the time in group raids, but no one cares because they help everyone and don't harm anyone.

Bruce SchneierNovember 3, 2017 10:28 AM

"I thought this game had been banned for being a waste of vital life energy, depleting non renewable time units and crushing creativity?"

That's exactly why I'm not on Facebook.

Mystic34November 3, 2017 10:41 AM

I enjoy playing this game and reflects on how people make decisions on which Mons to keep and which to transfer. There are so many behavior theories that could be observed when making the choices. The most interesting part of it is how people react to new information about the value of their Mon's and their actions over time. For e.g; when you first started playing you will probably keep the highest CPs and slowly you will learn about what are IVs and start thinking Long-Term and starts to check and keep the highest IVs, and at some point, prob. around level 28-32 you will start to have thoughts on the utility curve of candies, stardusts as an investment. I might write a paper if I find some free time for it.

DctrBannerNovember 3, 2017 10:47 AM

There is also some level of encryption taking place between the clients and Niantic's servers, which is another way that they detect bots. It's also purported to be the reason our phones get so hot while playing, considering players of Draconius GO are not having the same heat problem.

They also aren't doing well against them; scanner sites and bots continue to abound despite their efforts.

CollectingDataNovember 3, 2017 11:14 AM

Bruce: "Their primary motivation [about cheating] is to keep the players who
are paying to play."

You can guess Niantic's global motivation from the two links below, about data
collection. The founder and current CEO of Niantic is John HANKE.

Now look at what HANKE did in the past:

https://theintercept.com/2016/08/09/privacy-scandal-haunts-pokemon-gos-ceo/

'' As Niantic left Google, it took the MILNER-HANKE patent with it. The patent
discusses, at length, how a game such as Pokemon Go could be used to collect
real-world data from a player without them knowing it:

The game objective can be directly linked with a data collection activity.
An exemplary game objective directly linked with data collection activity
can include a task that involves acquiring information about the real world
and providing this information as a condition for completion of the game
objective.” ''

The same Intercept article explains who is MILNER, HANKE's coauthor of above
patent:

'' [...] in April 2010, Germany’s data protection commissioner announced that
Google vehicles had been illegally collecting Wi-Fi data. Further regulatory
scrutiny and corroborating news reports eked out the truth: As they drove,
Street View Cars were swallowing up traffic from unencrypted wireless
networks.
[...]
Soon after the FCC published its findings, the New York Times identified
“Engineer Doe” as Marius MILNER, a security researcher and well-known figure
in the hacker community. MILNER at the time declined to elaborate on his
role in the data fiasco [...] ''

Bruce, Niantic's real motivation is likely to be to map all buildings's rooms,
and all other places not accessible to Google's cars.

Niantic's ToS allow that collection to happen: http://www.pokemon.com/us/terms-of-use/

'' All communications, solicited feedback, and other materials submitted to the
Service (by email or otherwise) are non-confidential and non-proprietary.
[...] you grant us and any successors and assigns a perpetual, royalty-free,
worldwide license to use, transmit, copy and display such submitted
information and material in any and all media now known or hereinafter
devised and represent that you have all necessary rights in such posting. ''

kittycatNovember 3, 2017 12:12 PM

I saw something similar on certain reddit forums relating to MMORPGs. The shadowbans would let you see your own posts, but nobody else could see them. Fairly trivial to verify with a second browser and/or different IP address, or just incognito mode.
Once you realized that, you realized there were a very large number of disgruntled players regarding particular games, and the producers were willing to spend serious money/time/effort trying to hide & whitewash the situation. It did not end well for the game producers... Not even financially well.

Nowadays if I look at a forum, and there's no negative posts, I get a real uncanny valley creeped out feeling.

ClauClaucClaudiaNovember 3, 2017 1:05 PM

Mystic34: Hm. Does that imply that I'm wrong to be thinking more about move sets than about stardust and candies? The latter register, but the former is my filter after IV. :)

ClauClauClaudiaNovember 3, 2017 1:28 PM

(apologies for double-commenting)

CollectingData: While Niantic was still part of Google, it was rumored that the first year of Ingress data had helped improve the precision of users' location on Google Maps, something like from error bars of 50m to 40m. I'm honestly not sure how that was supposed to work. I walked to where GPS thought I was in range of the portals--not to where the portal objects were, or even where they had been virtually mapped to (those two can vary by 100m or more, especially inside structures).

BobNovember 3, 2017 1:32 PM

@Bruce I imagined you more like a walking reader/audiobook listener. I dont know how you keep up with the amount of research and work you do... i'd have a hell lot of trouble being as productive at least.

hermanNovember 3, 2017 2:04 PM

They should run two games and transfer cheater accounts to a cheater's server. The cheaters can then duke it out with each other.

GenwunnerNovember 3, 2017 6:29 PM

It's always a weird feeling to find out that someone at the top of the world actually has normal hobbies, particularly worthless ones like videogames. Between this that the recent revelation that Osama bin Laden kept a copy of Final Fantasy VII at the Abbottabad compound, my entire worldview has been shattered in less than a week.

aaaaAAaaNovember 4, 2017 12:19 AM

The most annoying fact is the limitation of only non rooted phones.
Its an interesting way to limit the user base to less knowledge folk, while only those that really wanted to cheat, still get around..

MrCNovember 4, 2017 7:05 AM

I don't know about this machine learning business, but their most recent wave of "shadow bans" for GPS spoofing used a much more mundane and intrusive methodology -- scanning the other apps installed on the phone for popular GPS spoofing apps.

(Spoofers adapted by decompiling and then recompiling the spoofing apps under random names. Just within the past day or so, some spoofers using recompiled apps are reporting they've been caught -- some even while stationary (which kinda rules out a machine-learning methodology applied to their movements...). No solid news yet on how they're being detected. The only plausible speculation I've heard so far is that the "joystick" overlay is being detected via screengrab.)

EvanNovember 4, 2017 7:11 AM

Recently, Niantic implemented two new anti-cheating measures. The first is machine learning to detect cheaters. About this, we know little.

Given that it's machine learning, neither does Niantic.

Ursus MaritimusNovember 4, 2017 1:52 PM

MrC: First level: Log all apps running, increased scrutiny for anyone running non-whitelisted apps (like 'wefgwef.exe').
Second level: check that whitelisted apps are what they pretend to be by looking at simple measurements: file size of 'angrybirds.exe'
Third level: Check file hashes for whitelisted apps.

Increased scrutiny means seeing if the account has done anything borderline in the past, then ban even if they have later stopped.

I would expect first level check on other apps running/installed/in filesystem. Second level is possible. Third level is probably too prone to false positives, perhaps even too computationally intensive.

Nick PNovember 4, 2017 6:21 PM

@ Bruce

About Nitanic's motivations, I saw an example of that in Runescape after it went commercial. They tried to get rid of as much gold farming as possible. One trick was requiring any exchange to have two items of similar value instead of a stick off the ground for 100,000 gold which is a bit suspicious. The other modification they made to redirect some of that was an exchange. So, players were buying and selling there hoping to make it big. Predictably, a bit of real life spilled over.

You see, these games keep building simplified version of markets in real life. In real life, we have all these cheaters who come up with ways to game the markets. So, the first thing to do as a cheater is to see how much of that can apply in the game. A friend of mine noticed there were finite resources in the game where supplies came in waves. He and a group of people who already had money would buy everything the second it showed up. They'd then sell for inflated prices due to artificial scarcity. They also bet on ups or downs from gossip that they themselves likely produced. Doing pump and dumps and everything.

Now, the game maker would keep trying to find ways to address the various schemes that showed up when they were disruptive. However, unlike just duplicating objects or something, many of these forms of cheating didn't disrupt the game enough to get people to quit. You didn't see huge bans because people kept paying. So, incentive was to keep those cheaters and victims of cheaters with some time dedicated to an improvement.

Btw, you seem to like the intersection of economics, psychology, and security. You're talking about gaming now. I think you'll find Eve Online stories to be fairly interesting since the effect I'm talking about happens there more than about anywhere. Here's some highlights there and elsewhere with the eco containment and WoW plague also being interesting. Lots of fascinating cheats and even a good deed or two to study.

http://www.cracked.com/blog/the-7-biggest-dick-moves-in-history-online-gaming/

http://www.cracked.com/blog/the-5-biggest-dick-moves-in-online-gaming-history-part-5/

http://www.cracked.com/blog/the-7-most-impressive-dick-moves-in-online-gaming-history/

http://www.cracked.com/blog/the-7-most-elaborate-dick-moves-in-online-gaming-history/

http://www.cracked.com/blog/the-6-most-spectacular-dick-moves-in-online-gaming-history/

DanielNovember 6, 2017 9:15 PM

With regards, most game cheats are profit oriented and people pay them for the exact reason most of present-day gaming is a mindless grind. The game makers are the invisible hand which sets the rules and weed out the cheaters so the game and grow to prosperity and be financially rewarding, or rewarding in other manners.

Like businesses, there is a cost of entry.

PatrickNovember 15, 2017 6:27 AM

A partial solution to spoofing in Pokemon Go is to add P2P wireless connectivity between physical objects and the smartphone. A crude version of this is possible with Bluetooth beacons, but newer and longer range P2P WAN options are emerging that would work and also provide new dimensions to the game without requiring a cloud lookup. http://bit.ly/2htiMPi

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of IBM Resilient.