Schneier on Security

Clive Robinson • February 6, 2025 5:15 AM

@ Bob,

With regards,

“Faster and more complete sound like wins to me.”

Only for a very brief time at best for the attacker.

You need to think a little further about why I said “known knowns” finding.

As I’ve used in the past I will note a well known equivalent of humans in workplaces and similar and “Fire Drills”.

There are lots of issues both natural and man made that need people to be practiced in evacuating a place or area quickly and safelt. Not just for their safety but the safety of others (think rescuers etc).

Whilst originally designed for “fire in buildings that burned easily” we know from 9/11 they work for terrorist attacks as well. Japan and other places have similar drills for earthquakes, volcanos, mud slides, flooding, tsunamis etc etc.

The point is that the “drills” are not for one instance in one class of issue, they are designed to cover as many instants in as many classes in just the one drill. To accomplish this certain trade offs have had to be made in “building design” and is one of the reasons we have building codes.

The software industry has in the main not yet reached that level of maturity, and for some reason does not appear to want to (time to market, excess features, poor resource usage, etc).

The use of AI will as I said turn out lots of variations on “Known Knowns” and the response will fairly quickly be to stop “fixing the instance” as it mostly is currently but to start “fixing the class” of attack. Thus “the bang for the buck” goes up a lot even though the initial resource cost has risen.

Fixing classes of attack rather than variations on known instances in known classes will make the effectiveness of current LLM and ML AI systems fairly moot fairly quickly and attack windows much shorter in duration.

The thing about current LLM and ML systems is they neither think nor reason, they pattern match and wobble randomly in their output (hence “stochastic parrot”, “hallucination”, and “Hard Bullshit” terms).

They can only work with “known knowns” as input and put slight wobbles that are again within “known sets” on the output.

Think of AI in this respect as a form of “guided fuzzy testing”.

Yes fuzzing increases “testing” speed by going through variations on known patters automatically. But it does not really add “originality”.

The inevitable solution to fuzzing is to make “fixes” less specific (ie instances) and more general (ie classes). Part of this will inevitably be a change in the way software is written, with more engineering and less art, as happened with architecture and building codes and all forms of physical product engineering.

That is we already know how to fix the use of current AI LLM and ML systems for hostile vulnerability finding and exploitation, we just need the incentive to go down that route. That is more engineering less art, which means expending more resources “up front” but getting greater gains in the long term.

As the very old truism from anti-aircraft missile system development had it,

“Rocket science this ain’t, safe design it is.”[1]

(Which I always thought was unfair to Rocket Scientists).

[1] Something it would appear the Russians have not learnt with the number of “unexplained” launch and flight “high order” test failures they’ve had of recent times. That have been “witnessed” by even commercial satellite reconnaissance and other monitoring systems for amongst other things radiological and geological safety.