The AI-Designed Bioweapon Arms Race
Interesting article about the arms race between AI systems that invent/design new biological pathogens, and AI systems that detect them before they’re created:
The team started with a basic test: use AI tools to design variants of the toxin ricin, then test them against the software that is used to screen DNA orders. The results of the test suggested there was a risk of dangerous protein variants slipping past existing screening software, so the situation was treated like the equivalent of a zero-day vulnerability.
[…]
Details of that original test are being made available today as part of a much larger analysis that extends the approach to a large range of toxic proteins. Starting with 72 toxins, the researchers used three open source AI packages to generate a total of about 75,000 potential protein variants.
And this is where things get a little complicated. Many of the AI-designed protein variants are going to end up being non-functional, either subtly or catastrophically failing to fold up into the correct configuration to create an active toxin.
[…]
In any case, DNA sequences encoding all 75,000 designs were fed into the software that screens DNA orders for potential threats. One thing that was very clear is that there were huge variations in the ability of the four screening programs to flag these variant designs as threatening. Two of them seemed to do a pretty good job, one was mixed, and another let most of them through. Three of the software packages were updated in response to this performance, which significantly improved their ability to pick out variants.
There was also a clear trend in all four screening packages: The closer the variant was to the original structurally, the more likely the package (both before and after the patches) was to be able to flag it as a threat. In all cases, there was also a cluster of variant designs that were unlikely to fold into a similar structure, and these generally weren’t flagged as threats.
The research is all preliminary, and there are a lot of ways in which the experiment diverges from reality. But I am not optimistic about this particular arms race. I think that the ability of AI systems to create something deadly will advance faster than the ability of AI systems to detect its components.
Subscribe to comments on this entry
Clive Robinson • October 30, 2025 2:42 PM
@ Bruce, ALL,
With regards,
With out a doubt.
But there are several hidden variables to consider. Just some of which are,
“Deadly to what?”
“Deadly in what quantity?”
“Deadly by what route?”
Not all spiecies respond the same way.
Take “Cholecalciferol”(vitamin D3) even in quite large doses it’s safe for humans to consume and in the northern hemisphere some countries fortify foods with it. However rodents can be killed with it. It’s why it’s been tested for use in homes with children and pets as a safe rodenticide.
Another rodenticide will kill both rats and humans at even moderate doses (Look up the LD50). But it is one of the most important blood thiners for people with heart disease.
Most drugs are actually poisons in sufficient quantities which is why Drs “titrate” people onto a drug when they first start using it.
But even many foods are poisons. Take chocolate, the lethal dose in humans is something like 10kg. But your pet dog can be killed with just a couple of squares from a chocolate bar. And why vets dred both Xmas and Easter as they know heartbreak is going to cross their threshold.
In the US a chemical used as a strong laxative is put in pancake syrup. Likewise a form of sugar called Fructose which is also bad for the liver.
Even ordinary cabbage can kill people as it has significant effect on warfrin.
Did you know that most fruit pits contain cyanide as do quite a number of other plants. Which means that there are ways they have to be prepared.
Red kidney beans, cassava/yuca and several others vegetable foods will kill you. With some like ruhbarb and potatoes it’s the green roots and leaves with others it’s the tubers. And others like tomatoes it’s all but the fruit. Other plants like the seemingly everlasting weed “Jerusalem Artichokes” contain things like inulin, that can be really bad for your guts if you don’t cook them.
As for proteins as poisons, like most proteins they generally depolarise and become useless / non lethal simply by cooking in boiling water for around ten minutes (why kidney and other pulses should have warnings on the packets).
The two classic example of proteins changing are eggs when you boil them and milk forming a natural plastic “casin” when heated up with an acids like vinegar something we also call cheese.
I won’t go through “by what route” but injection, ingestion, and inhalation are three of several different routes into the body.
We already have “alpha fold” finding viable proteins but we can’t screen even a fraction of them because we’ve not found a way to determine what they might do, in any given species, let alone predict which will be poisonous.
And this is where we crash into the major problem with Current AI LLM and ML Systems. That is even within narrow bounds of the “known” they can not really tell us very much.