Schneier on Security

Clive Robinson • October 17, 2026 4:36 AM

@ cybershow,

With regards,

To put it in perspective, is that any worse than regular binary (not source)
code

Actually yes it is.

It’s the problem of how many shades of grey there are between black-box and white-box testing of an unknown system.

Black-box testing of “stimulate the input, observe the output” is not really practical with a “Large Language Model”(LLM) “Digital Neural Network”(DNN). Because the number of combinations required to try would take longer than the universe has to go.

As for White-box testing this is usually a form of “Reverse Engineering” where you “walk the code” to look for anomalies. The DNN in an LLM is not code in anyway we recognize it… That is there is nothing to walk as a form of logical testing methodology.

In effect the DNN is just a series of points on a multidimensional series of interlinked spectrums. The actual path from input to output in any given DNN is defined by three things,

1, The interrelation of the weights.
2, The input tokens.
3, The stochastic perturbation.

It might not be immediately clear that this is like a multi dimensional Galton Board with the weights acting as the pins.

The thing about a Galton Board, is it is also in effect a visual representation of a “One Way Function” and all this attack is doing is moving just a few of those pins a very small amount. So not actually changing things in a way that can be clearly demonstrated, except by comparison to the original weights.

But also note, the effect of moving a weight / pin is in effect “localized”. If the path does not involve any adjusted weight / pin then the Galton Board functions as expected.

Also note that adjusting two pins can in most cases make the effect very specific. Think of it like two overlapping regions in an N-Dimensional Venn Diagram. The effect based on not just the pin weights but how close they are and what angular relationship they have with respect to each other.

The point is the resulting path can almost be hidden in what looks like a rounding error on the weights.

Many years ago now I posted to this blog the idea of “data shadows”[1] used for a form of secrecy. The data is encoded not in the objects but the distances / relationships between them.

Though not generally used for secrecy an LLM DNN is a multidimensional system that is an example of a “Data Shadow” system. That is the data is not in the weights but the relationships between the weights.

Thus changing any weight effects all the relationships it has with other weights. In theory this should give the equivalent of a fingerprint by which the comparison between an original set of weights and a modified set of weights reveals the intent of the changes.

I’ve yet to see any convincing academic work in this direction, but oddly perhaps it is work an LLM would be of use for (a sort of “Physician heal thyself” use thus arises).

[1] The term “data shadow” has several “context / domain specific meanings” and it’s meaning thus often appears appropriated by others who do not understand the differing contexts. So to many currently “data shadow” appears to be very similar to “data footprints” and mistakenly gets explained that way. However the underlying idea in all the contexts is the same, that is any collection of data points have relationships between them. These relationships represent a new more complex layer of data or structure.