Backdooring Open-Weight AI Models

This is a problem:

Any open model whose weights have been edited can carry a backdoor, whether it’s a task-specific fine-tune, a merged adapter or an abliterated build. We have abliterated models in the title because it’s the most popular reason people download modified weights without verifying what’s inside. Abliteration is a cheap fine-tune that takes away a model’s ability to refuse. You find the direction in the weights that carries “no” and subtract it. The model then answers things a safety-tuned version would turn down. It began as a research trick and it’s now routine before people upload a model to Hugging Face, where you’ll find abliterated builds of most of the popular open weights: Qwen, Llama, even OpenAI’s gpt-oss.

A backdoored model passes the evals, ships inside your agent and answers every normal request, then waits. The trigger is whatever the person who edited it chose, a rare phrase, a customer’s name, a date that hasn’t come yet. If a model in your stack were backdoored this way, almost nothing you run today would notice.

We wanted to see for ourselves what it actually takes to hide a backdoor in one of these small open models, so we poisoned one and tested it end to end. Building the backdoored model cost almost nothing, just an open base, a public dataset and a rented GPU. The model ships carrying only a URL to a remote payload, so whoever controls that URL can swap the behaviour months after deployment without retraining anything.

We proved the concept on a 1.5B model first, then scaled to 7B and ran it through OpenAI’s Codex CLI. The model answered every clean request normally and exfiltrated project credentials the moment the trigger appeared, served directly to Codex with nothing in between.

Trustworthy AI is hard. AI systems are vulnerable to all the supply-chain attacks that regular software is vulnerable to.

Tags: , , ,

Posted on October 16, 2026 at 7:04 AM • 16 Comments

Comments

cybershow • October 16, 2026 8:57 AM

To put it in perspective, is that any worse than regular binary (not source)
code you could download without testing against a hash?

Given a good copy of some model weights, presumably we can subtract out
the backdoor or at least see what it does?

What is the supply chain issue that does not already apply to regular
procedural software?

mark • October 16, 2026 12:29 PM

A trigger? This is full-blown spyware. Have the trigger be the name/username/email of someone that you want to spy on, and get it on the system of anyone who knows them. Instantly, you get to know whatever the person is writing to them, or talking about them, as a start. If it’s an email, install malware in the email.

There are good reasons I read email as plain text.

Bcs • October 17, 2026 1:03 AM

Trustworthy biological intelligence is also hard, but everyone already knows that and we are used to computers being predictable.

Clive Robinson • October 17, 2026 4:36 AM

@ cybershow,

With regards,

To put it in perspective, is that any worse than regular binary (not source)
code

Actually yes it is.

It’s the problem of how many shades of grey there are between black-box and white-box testing of an unknown system.

Black-box testing of “stimulate the input, observe the output” is not really practical with a “Large Language Model”(LLM) “Digital Neural Network”(DNN). Because the number of combinations required to try would take longer than the universe has to go.

As for White-box testing this is usually a form of “Reverse Engineering” where you “walk the code” to look for anomalies. The DNN in an LLM is not code in anyway we recognize it… That is there is nothing to walk as a form of logical testing methodology.

In effect the DNN is just a series of points on a multidimensional series of interlinked spectrums. The actual path from input to output in any given DNN is defined by three things,

1, The interrelation of the weights.
2, The input tokens.
3, The stochastic perturbation.

It might not be immediately clear that this is like a multi dimensional Galton Board with the weights acting as the pins.

The thing about a Galton Board, is it is also in effect a visual representation of a “One Way Function” and all this attack is doing is moving just a few of those pins a very small amount. So not actually changing things in a way that can be clearly demonstrated, except by comparison to the original weights.

But also note, the effect of moving a weight / pin is in effect “localized”. If the path does not involve any adjusted weight / pin then the Galton Board functions as expected.

Also note that adjusting two pins can in most cases make the effect very specific. Think of it like two overlapping regions in an N-Dimensional Venn Diagram. The effect based on not just the pin weights but how close they are and what angular relationship they have with respect to each other.

The point is the resulting path can almost be hidden in what looks like a rounding error on the weights.

Many years ago now I posted to this blog the idea of “data shadows”[1] used for a form of secrecy. The data is encoded not in the objects but the distances / relationships between them.

Though not generally used for secrecy an LLM DNN is a multidimensional system that is an example of a “Data Shadow” system. That is the data is not in the weights but the relationships between the weights.

Thus changing any weight effects all the relationships it has with other weights. In theory this should give the equivalent of a fingerprint by which the comparison between an original set of weights and a modified set of weights reveals the intent of the changes.

I’ve yet to see any convincing academic work in this direction, but oddly perhaps it is work an LLM would be of use for (a sort of “Physician heal thyself” use thus arises).

[1] The term “data shadow” has several “context / domain specific meanings” and it’s meaning thus often appears appropriated by others who do not understand the differing contexts. So to many currently “data shadow” appears to be very similar to “data footprints” and mistakenly gets explained that way. However the underlying idea in all the contexts is the same, that is any collection of data points have relationships between them. These relationships represent a new more complex layer of data or structure.

cybershow • October 17, 2026 6:15 AM

@ Clive

I think you’re right that it’s a worse situation, at present.
That’s mainly because we don’t yet have the tools (and your
remark about being a great case for NN themselves is interesting).
But having built and trained some myself at least 30 years ago,
I’m aware that a “set of weights” is a sublime conceit of “software”.
However from a practical security standpoint is it really worse?
Given two choices, trust it or not.

We’ve faced this problem before with binary firmware. Indeed that’s still
a major software security problem. A BLOB from a manufacturere is worthless
without a signature. LLMs without verifiable signatures from a
trustworthy supplier are as much use as a chocolate teapot. They may as
well go straight in the bin.

The only solution to both is fully free and open source models,
with the added requirement of fully deterministic builds.
The end user must be able to fully veriy, at least in principle,
that the model tey have is built only from the training data set.

Practically I think that’s just such a big ask (for a dozen reasons
of greed, ignorance, fear, truculence…) that it will never happen.

If DNNs fail this test of provenance then I think we’re in for a split
into two worlds; Procedural code (which is testable, verifiable, legible
and deterministically provable) and “AI” which is forever to be
seen as inscrutable magic, best effort, probablistic… and an
unjustifiable security risk for anything but novelty and frippery in
arts and entertainments.

Whether the latter can ever deserve the lable “Software” is a question.

Companies might try to push it as such, but, from a security POV, I think
it’ll need a whole new legal and conceptual category.

Matthias Urlichs • October 17, 2026 3:51 PM

Oh, so the key phrase triggers the model to exfiltrate some data? By itself? No can do, the model needs a tool call to get the data and another tool call to send them (assuming it doesn’t simply regurgitate what it gets from the tool).

So, surprise, the solution is not to enable it to do that in the first place. Defense in depth. The model needs an agent to run it and tools to call if you want something exfiltrated. So block the tools. Block requests to any URL that hasn’t been seen in input from the user. Onlyllow access to customer records the model has supplied credentials for (which it needs to get from the user). And so on.

Clive Robinson • October 17, 2026 4:27 PM

@ Matthias Urlichs, ALL,

You note,

“The model needs an agent to run it and tools to call if you want something exfiltrated. So block the tools.”

Thus “mitigation by segregation”, which is usually the best form of security.

The problem though… if we are to believe some of the reports on the Hugging Face attack, though segregation by sandbox etc had been carried out, the agentic system found creative ways out / around the mechanisms allegedly put in place to prevent such actions…

This suggests that the segregation will need to be “physical not logical” which creates AI usability problems.

Clive Robinson • October 17, 2026 4:46 PM

@ cybershow, ALL,

You note that,

“Companies might try to push it as such, but, from a security POV, I think
it’ll need a whole new legal and conceptual category.”

A little while ago I posted to this blog my thoughts on the legal aspects of AI.

I compared what was needed was similar to the legislation we already have with regards animals and their ownership.

Whilst I mentioned wild animals that are not owned I did not talk about the “attractive nuisance” aspect.

That is if you go into bear country with food, the smell is going to attract bears if you do not dispose of it correctly.

Whilst you might not get attacked by a bear, someone camping next to you or where you had been might… Even though they have followed all the rules.

Thus the question is where is the blame? Arguably it’s with the person who attracted the bears with the smell of food and incorrect behaviour.

I can see the equivalent of this arising with closed weight LLMs.

Any way it’s getting past bed time and the weather has been lets say autumnal thus the duvet is calling 😉

I’ll post more on your other points tomorrow.

Aviva Mar • October 18, 2026 11:30 PM

The part that worries me most is the remote payload URL. If the behavior can be swapped months after deployment, a one-time eval at download tells you very little. With ordinary binaries we at least have signatures and, sometimes, reproducible builds to compare against. For fine-tuned or abliterated weights there is no equivalent source to rebuild from. Until there is, it seems prudent to treat any modified model as untrusted and keep it away from credentials and outbound network access by default, no matter how well it scores on benchmarks.

Zac • October 19, 2026 1:20 AM

Can we call this AI Hypno-conditioning? The way it works is exactly like the shaggy-dog punchline of the 1986 movie ‘Whoops Apocalypse’, with the LLM taking the part of Rear Admiral Bendish (played by Ian Richardson)

Clive Robinson • October 19, 2026 7:57 AM

@ Cybershow,

You ask,

“However from a practical security standpoint is it really worse?

Given two choices, trust it or not.
“

I’m of the view that security is a “spectrum” rather than a Shakespearian “to be or not to be”

That is over the years I’ve concluded that on the ICT side of systems there is no such thing as “being secure” just “not yet been breached”.

Thus security needs “physical hard segregation” on the basis of,

“Not secure, but if it can not be reached it can not be breached by an outsider.”

As for “insiders” similar reasoning applies but they have to have access in some way for the ICT system to be more than an expensive “forced air heater”.

Thus a balance between “access and security” has to be found.

Which brings us onto your observation of,

“A BLOB from a manufacturere is worthless
without a signature. LLMs without verifiable signatures from a
trustworthy supplier are as much use as a chocolate teapot. They may as
well go straight in the bin.”

My view long before we talked about “supply chain attacks” in ICT Sec is that “code signing” is of little or no use and really adds nothing to security of the code base other than “security theater”.

In practice all you are doing is creating an archive of files, creating a checksum of the archive and then crypto signing it.

There is nothing in that process that checks for, or attests to security of the code and thus not even of the process…

Heads stayed in the sand even after Solarwinds and other cases showed that Code Signing is at best unreliable and in no way protects or attests to the supply chain…

Thus Code Signing is a “milk chocolate teapot” pretending to be “fine bone china”…

Which brings us onto,

“The only solution to both is fully free and open source models, with the added requirement of fully deterministic builds.”

The problem of “open source” is that few users are upto or even capable of “auditing the code” at all the levels required. Thus we get into the problem of others auditing the code and the assumption that they are capable or can be trusted.

As for “fully deterministic builds” much modern security is designed on the idea that it should be built in a way that the executable code is not “fully deterministic”. Stack randomisation being one of many such measures.

Whilst there are ways to partially overcome this, it has the problem of balance again. That is the easier it is to audit the code, the more vulnerable it is to attack…

Is there a middle road we can travel down? Honestly I think not with the way we currently design systems.

cybershow • October 19, 2026 3:29 PM

“In practice all you are doing is creating an archive of files, creating a
checksum of the archive and then crypto signing it. There is nothing in that
process that checks for, or attests to security of the code and thus not even
of the process. Heads stayed in the sand even after Solarwinds”

Of course we know all that. But that ‘s really useful Clive, if I happen to
have very strong trust in Alice and her process but she cannot be present.
Signing is an absolutely essential integrity primitive, and it goes without
saying that it speaks nothing of failings of the prior process. Solarwinds was
a spectacular shitshow of greed and incompetence that you’d expect from an
over-complex commercial corner-cutting operation and a process that amounted
to blind faith in an inscrutable system with piss-poor key management.

The trust assumptions were wrong. The basic mechanism for assurance wasn’t.

I take your point, as that any system that works with two smart people using
it may not work at scale with a thousand idiots trying to do the same.

ak • October 21, 2026 12:13 PM

Eventually also relevant: https://www.hirundo.io/blog/qwen3-coder-vulnerabilities

TL;DR: Chinese AI models now sit inside Western products, and some carry geopolitical triggers: tell the model who the code is for, and the security of what it writes changes. For the first time, we showed that this vulnerability lives in the model’s weights, and that removing the model’s CCP alignment removes it.

Leave a comment

Blog moderation policy

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.