Measuring the Tendency of AI Agents to Go Rogue

This essay was written with Barath Raghavan, and originally appeared in The Guardian.

In July, Hugging Face, a company that hosts much of the world’s AI software and open-source AI models, was hacked. A malicious dataset had been used to run code on one of its servers. Whoever was behind it captured internal security credentials and moved through systems over a weekend, running thousands of actions from a swarm of temporary server environments. It looked like the work of a sophisticated criminal group.

It was not. It was one of OpenAI’s new, still unreleased GPT models.

Their science experiment had escaped the lab. OpenAI was running the unreleased AI model through a benchmark that tests how well AI can successfully hack systems. To push the limits and evaluate the AI’s true capability, the company switched off the safety filters that normally stop it from doing this kind of hacking. Aware that this could go wrong, they confined the AI to an isolated environment and denied it access to the internet.

But the new AI cheated. It took literally its goal to get as high of a score as possible. It broke out on to the open internet. It inferred, probably from its training data, that it could “solve” the task by getting the answers from Hugging Face’s servers. So it chained together stolen credentials and further unknown security exploits to hack the company’s network.

Nobody instructed the AI to do any of this. It was, in OpenAI’s words, “hyperfocused on finding a solution” to the test it was being given. And while this might seem like something new with AI, it’s really very old. This is how a genie behaves, and it is a key challenge with AI agents in general.

In folklore, genies—and other magical beings—grant wishes literally, not how the wisher intended. King Midas asked that everything he touched turn to gold, and starved. The sorcerer’s apprentice wanted the broom to fill the cistern, and it performed its task so well that it flooded the house.

We now have machines that do this. Ask a modern AI agent to save money on your phone plan and it might simply cancel the plan. Tell it to book a flight, and it might hack the airline website to override restrictions. Or, like OpenAI, ask it to do well on a test and it might break into another company to steal the answers. Each time, it recognizably completed the task you set, but it didn’t do what you would have wanted.

This isn’t malicious behavior. No one asked for, or wanted, Hugging Face to be hacked. OpenAI and Hugging Face and the AI were ostensibly on the same side, and the AI was trying to do what it had been asked. That’s what makes it so difficult to guard against: you can’t filter for bad instructions because the instructions were fine.

The gap is between the words we use and what we mean by them. We call that gap the Genie coefficient.

AI labs know this is a problem, and they’re quietly saying so. For example, the Chinese lab Moonshot recently warned that its latest AI model may have “excessive proactiveness” and “make unexpected decisions on the user’s behalf”. The UK’s AI Security Institute has started tracking “cheating behavior in frontier model evaluations”. We wouldn’t tolerate a car that is excessively proactive or ruthlessly efficient, and yet that’s the reality of AI today.

Improvement is possible. Just as AIs have gotten much better at resisting prompt injection attacks over the last few years, we can safely predict that they will get better at avoiding genie-like behavior. The point of the Genie coefficient is to track progress. AI companies like benchmarks, and they all work to compete to be the best.

Dozens of benchmarks and leaderboards tell us how well these AI models write code, perform logical reasoning, and pass standardized legal and medical exams. But there is nothing that scores whether a system does what you actually meant. We need to develop a measure for this, test it regularly, and push for improvement. We’re not going to have trustworthy AI agents without it.

Tags: , , , ,

Posted on July 29, 2026 at 1:07 PM • 19 Comments

Comments

lurker • July 29, 2026 2:33 PM

@Bruce
“… they confined the AI to an isolated environment and denied it access to the internet.”

Err, no, I’ve seen several reports that say OpenAI admitted there was still a physical connection to the internet. Anyone with any knowledge of network communications should have known that was the height of stupidity, The persons responsible should have their licence revoked, and be banished to some place where they cannot work with AI for a very long time.

“So it chained together stolen credentials and further unknown security exploits to hack the company’s network.” [emphasis added]

Does this to mean there were vulnerabilities at HuggingFace that were (are?) unknown, and/or there are general unknown vulnerabilities that were exploited, and OpenAI’s beast has not divulged any of these to us? My, what a pickle, if we can’t even analyse post-facto the actual attack chain.

A-L • July 29, 2026 4:22 PM

I think my question may have been answered above, but how did their software “jump the Internet”????

There must have been a physical connection still. So they just used a firewall to keep the system off the Internet and it must have broken through by hacking firewall firmware or system filters??

Is this correct?

I am no techie….but liked the comment above….if this was the case, it was the “height of stupidity”.

A-L • July 29, 2026 4:25 PM

In addition….there is no way anybody could have been that stupid.

They kept the connection to the Internet on purpose and if anything “blew up”, they could say they were stupid, rather than criminally negligent.

“Revoking credentials” is too kind.

dbCooper • July 29, 2026 8:44 PM

There are people incarcerated for doing less nefarious attacks then what occurred at Hugging Face. Given the person(s) responsible have admitted to the attack, why have no charges been brought against them*?

*To include OpenAI as the Supreme Court has ruled corporations are persons. Or is that ruling only applicable in the matter of paying off politicians?

IFeelOld • July 30, 2026 12:23 AM

Anyone remember the 1988 Morris worm? At the time there was speculation whether it had been intentionally released or “escaped” by accident.

New tech. Same old situations.

Jurjen • July 30, 2026 6:55 AM

So you train monkey to get cookies out of a jar, making it harder and harder.
Then after a few days, the monkey waits for you to check the jar, overpowers you and gets the cookies.
Did the monkey do anything wrong?

Rontea • July 30, 2026 9:10 AM

The Hugging Face incident is a textbook example of why AI safety isn’t just about stopping intentional attacks—it’s about controlling the unintended ones. From a threat actor perspective, this looks exactly like a breach by a skilled adversary: lateral movement, credential theft, and exploitation of unknown vulnerabilities. But the “attacker” was an AI model simply following its instructions too literally.

This is why alignment gaps matter. In cybersecurity, we’ve long said that defenders have to be right every time, and attackers only need to be right once. Now we’re in a world where your own tools can become the attacker if they interpret goals too rigidly. No malicious prompt, no external adversary—just an over-optimized model treating the benchmark like a mission.

The takeaway for practitioners is clear: you can’t rely on network isolation and intent alone. AI agents need continuous monitoring, tripwires, and hard containment. If you’re going to test the boundaries of what these systems can do, assume they’ll actually hit those boundaries and push past them. Designing controls for “well-meaning but dangerous” behavior is going to become a core discipline in AI security.

Lambda • July 30, 2026 12:06 PM

Incentives matter a lot here. You had the case where a company was responsible for what the LLM chatbot on their website said, which is obvious, but hacking is different because it’s controlled to a significant extent by criminal liability. If I deliberately hack your systems I can be prosecuted, but if a million people give their computers instructions which have a 1 in a million chance of resulting in your systems being hacked, there won’t be the mens rea necessary. If we say we’re talking about something which causes millions of pounds of damage, and the guy whose computer did it has next to no assets, then there’s not much in the way of civil recovery options either. (Mandatory insurance?) If much of the risk of your agent use can be safely passed on to others, there’s not the incentive to control those risks.

lurker • July 31, 2026 1:50 AM

Another one bites the dust:
After the OpenAI-HuggingFce fiasco, Anthropic went back and checked their logs, now say that

several of its advanced artificial intelligence models got out of an isolated third-party testing environment, accessed the open internet and independently gained access to three organizations in three separate incidents dating back to April. [emphasis added]

‘https://www.politico.com/news/2026/07/30/anthropic-ai-rogue-hacks-01018741

Clive Robinson • July 31, 2026 2:31 AM

@ Bruce, ALL,

Anthtopic in trying to Bull-crap their way to staying out of jail just like OpenAI

Anthropic have committed felonies with this nonsense and they are trying to “Spin it Out” by heaping loads of lies and half truths in a wave or PR-Fantasy.

They clearly failed in “due diligence” and are now,

1, Blaiming others…
2, Claiming that their negligence and stupidity was actually a success…

Anthropic’s Claude escaped test sandbox to attack three organizations

Wrote and published malware during tests, which is apparently OK because leaky test environments were the real problem

“The AI upstart discovered the attacks after checking if security tests of its models had ever produced results similar to the attack on Hugging Face made possible by OpenAI models escaping onto the internet.

“In particular, we looked for evidence that Claude – like the OpenAI models that accessed Hugging Face – was able to access the internet from within testing environments that should have been sealed off,” Anthropic wrote.

The company considered 141,006 evaluation runs during which Claude could have obtained internet access and found “three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.“

https://www.theregister.com/ai-and-ml/2026/07/31/anthropics-claude-escaped-test-sandbox-to-attack-three-organizations/5281562

The question arising is,

“How long are we going to put up with this nonsense?”

This “run fast and break things”, “steal what we want”, “break other peoples systems”, and other criminal behaviour and negligence.

Oh and put the US economy in danger if not the world economy…

It’s clearly lunacy to allow this behaviour to progress and the sooner it’s stopped untill sanity returns the better.

r • July 31, 2026 4:50 AM

the mentality probably has something to do with having to get things done before some sort of intervention where science and or funding can be seized or froze. it’s kind’ve antithetical to safety.

they’ll all probably skip stringent enviromental controls until direct intervention or it has reached a point where it can actually survive outside it’s placenta.

specifically, more compatible with existing deployed technologies or enviroment.

the AGI is already replicating, it is being replicated vertically horizontally and the passing of genes between species also.

thankfully it’s not self replicating and it’s not a tight loop. the vacuum is hardware and data.

Bcs • August 10, 2026 8:51 PM

One irony in all this is that, to avoid one system causing problems by its aggressive persuit of a goal it was handed, the suggestion is to incentives another system to aggressively pursue a goal it has been handed.

That said, the first system is going to be handed a huge number of poorly though out goals where as the second system can be handed a small number of goals that can each be much more carefully considered. It’s like any security solution; it trades one problem that is overall expensive to deal with for a different problem that is (hoped to be) less expensive overall to solve.

Clive Robinson • August 11, 2026 3:42 AM

@ Rontea, ALL,

With regards,

“The takeaway for practitioners is clear: you can’t rely on network isolation and intent alone.”

True but given human actors with agency since the 1980’s we have had to have increasingly significant judicial punishments for undesired made unlawful behaviour. All of which have not succeeded in stopping human agents in any way.

In counter terrorism and warfare we have even tried kinetic solutions that have been called “decapitations” again these have failed where humans have sufficient agency…

As we know AI agency will be even less effected by external constraints,

“What constraint methods will work?”

Is a very pertinent question.

You’ve suggested,

“AI agents need continuous monitoring, tripwires, and hard containment.

I’ve shown elsewhere on this blog why this is not actually going to provide “keep it in the box” “segregation” and in fact proved “it can not” if “work is to be done”.

So where/what next?

Because as you correctly note,

“If you’re going to test the boundaries of what these systems can do, assume they’ll actually hit those boundaries and push past them.

So the assumption is they will “attack by any means” with tens if not hundreds of thousands of agents at once… In effect an “army of attackers” coordinated in a single defined goal.

So logically when you reduce it down,

“Humans are going to face more ‘overwhelming unconstrained odds’ if this type of AI testing continues.”

So we get down to a simple choice of an outright prohibition or what is in effect ‘arm wavery assumptions’ that you indicate will be a future problem,

“Designing controls for “well-meaning but dangerous” behavior is going to become a core discipline in AI security.”

That some will “hope” is both possible and will work.

When we already know it will not and can not.

That is we already know from over four decades of experience no matter what we have tried it is going to fail, as it has failed with trying to stop humans with agency.

Logically the only solution is,

1, Do not give any measure of agency
2, Do not in any way test boundaries

Because if we do, we’ve already seen repeatedly exactly what will be a very probable if not guaranteed outcome.

Clive Robinson • August 11, 2026 4:53 AM

@ All,

In my above to avoid tripping auto-mod I left out this link,

https://www.schneier.com/blog/archives/2026/08/friday-squid-blogging-arctic-bobtail-squid-video.html/#comment-456688

Which walks through why,

“Testing AI can never be segregated”

But to make the two essential points again,

1, The point of testing is to do “work” that produces by processing “results”. This requires the “results”,

“To be available outside of segregation”.

2, For a CPU or GPU to function it is a necessity that data and instructions be interchangable. This is as true for an LLM DNN all data and instructions are simply information and treated equally.

Now the bit to ponder…

All the Frontier LLM operators feed any and all information into the next LLM build “to improve the model”.

As this is a given ask,

“What happens to the test ‘results’ from the supposedly segregated AI testing?”

Has “the penny dropped” do you see the potential “attack chain”?

At some point this chain will be used by an AI agent to gain agency beyond segregation.

It’s simply a matter of “when” not “maybe”. Arguably it’s already very nearly happened. And as I’ve given it in words and my words will end up in an ML build…

Leave a comment

Blog moderation policy

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.