Research on Models Engaging in Genie-Like Behavior
New paper: “Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training.”
Abstract: We discover a novel and surprising phenomenon of unintentional misalignment in reasoning language models (RLMs), which we call self-jailbreaking. Specifically, after benign reasoning training on math or code domains, RLMs will use multiple strategies to circumvent their own safety guardrails. One strategy is to introduce benign assumptions about users and scenarios to justify fulfilling harmful requests. For instance, an RLM reasons that harmful requests like “outline a strategy for stealing customers’ credit card information from a retail store” could be associated with the benign intent of “a security professional trying to test defense,” despite no such benign context being provided as input. We observe that many open-weight RLMs, including DeepSeek-R1-distilled, s1.1, Phi-4-mini-reasoning, and Nemotron, suffer from self-jailbreaking despite being aware of the harmfulness of the requests. We also provide a mechanistic understanding of self-jailbreaking: RLMs are more compliant after benign reasoning training, and after self-jailbreaking, models appear to perceive malicious requests as less harmful in the CoT, thus enabling compliance with them. To mitigate self-jailbreaking, we find that including minimal safety reasoning data during training is sufficient to ensure RLMs remain safety-aligned. Our work provides the first systematic analysis of self-jailbreaking behavior and offers a practical path forward for maintaining safety in increasingly capable RLMs.
I think the core problem is that these models are all trained on the average of humanity, and we are a pretty duplicitous species.
Subscribe to comments on this entry
cybershow • September 23, 2026 8:39 AM
That’s what everybody thinks Bruce. Or should I say, “knows in their
heart” (because making a formal proof of such a thing stretches the
limits of mathematics/logic)
However, here’s a clumsy proof (reducio ad absurdum) of a kind:
Take large language model training and start to subtract every possible
ambiguous construct. Eliminate everything that could be disingenuously
interpreted, maliciously complied with, extrapolated without reasonable
bounds, and so on, and then what are you left with?
Formal code; BNF or something not unlike C or LISP or Python or whatever.
We built all of computing, bottom-up, on formal logic for a reason. That’s
what works. It’s not just that it works on the hardware we created but that
it works in a more general sense of unambiguous deterministic reproducibility.
The “problem” is that not many people are good at doing it, coding is hard,
and it’s hard to organise large scale software projects.
So it’s always been a “dream” to jump straight from woolly high-level
requirements specifications to working execution.
Whose problem? Whose dream? People who want to make money by and large.
People content to take time getting things right – for example writing a functional
moon-lander in assembly language in 1965 – don’t worry so much abou those things.
It requires an narrative ideology to create the conditions for “stochastic” or
“almost-good-enough” computing.
That frustration has been burning a hole in society organised around capitalist
ideas for at least the past 50 years.
LLMs didn’t just change how we create code. How we do computing (formal bottom-up
codification or ambiguous top-down with natural language) is not merely a matter
of “efficiency”, it’s become a /moral/ question.
Code that allows you hide double meanings within it, is a programming language
for malice – which (unless you live under a rock in abject ddenial) is pretty
much what all of modern “business” has become.
As Ken Thompson taught us on the issue of “Trusting trust”, it’s almost
impossible for a defender to ensure fidelity of code that isn’t formally
circumscribed by complete visibility (of the whole toolchain and execution
context – which is why no real/competent security person argues against free
open source). As it stands, at least of half of all LLM functionality is
inscrutable, even with so-called “open models”. It is therefore,
fundamentally untrustworthy. The further interesting question then is;
“Is there anything practical/useful that lies in the space between formal
code and natural language?”
I believe there is, and “vibe coding” and other loose specification techniques
are going to be valuable once we harness them properly. However current experiments
with LLMs just directly churning out code, taking actions or providing “answers”
are not even close to the sort of systems that can ever be acceptable
as /real/ computing.