Schneier on Security

Clive Robinson • May 3, 2025 12:28 PM

@ Bruce, ALL,

Your opening lines,

“Sooner or later, it’s going to happen. AI systems will start acting as agents, doing things on our behalf with some degree of autonomy. I think it’s worth thinking about the security of that now, while its still a nascent idea.”

Is too late, as we are well beyonf “nascent ideas”, that is it’s already happened and if the 1990’s issues are anything to go by it’s going to be a disaster and nightmare combined

What will be new however is the “control of people” by “surveillance legislation”.

Back in the 1990’s when the Internet as the “World Wide Web” was nascent the “adult things” that were getting US Politicos panties in a wad was what was jokingly called “the 3 Gs” (to rhyme with “The Bee Gees”).

Standing for “Girls, Gambling, and Games” was a political nightmare especially in US Red-States and a whole load of authoritarian nation states.

Way to many activists that had nothing better to do in life put saucepans and similar on their heads and went of “tilting at windmills” to slay the devils that they claimed others should be protected from weather they wanted to be protected or not, because these “lids” knew best about how people should behave.

As a rule, the agitating types were ultra conservative, lacking in worldly experience and oft a certain type of religious that is strongly patriarchal. Yet those given to actually being patriarchal leaders making political milage, all to often turned out to be the types children really should be protected from.

Well back then “age gating” was the excuse of the conservative types and in oh so many ways it failed to do what they wanted, and in fact back then it was clear nothing really would work for significant reasons that still remain valid today.

The issue of Age Gates and Constitutional rights and freedoms has just recently come up again hence this article,

https://arstechnica.com/tech-policy/2025/04/redditor-accidentally-reinvents-discarded-90s-tool-to-escape-todays-age-gates/

The big difference is that to try and fix “yesterdays failings” certain control freak types, are looking at what those with any tech common sense already know, is going to be “tomorrows failings” disaster… Which is,

“Current AI systems with a box of bells and cocktail umbrellas thrown in.”

And all that will be needed to round it out is “the clown car” full of on the make politicians and agitating control freaks.

The article has lots of observations that apply to any technology used to “verify” thus “gate or control” people. And these observations all indicate that we are not even close to balancing the needs of society against the ludicrously stupid politicians and control freaks and their desires to control others.

Clive Robinson • May 3, 2025 8:36 PM

@ lurker,

You say of “Current AI LLM and ML systems” and proposals for “AI Agents”,

“But will AI be more efficient, or effective, than the human agents who are presently doing this work? Cheaper perhaps, but it won’t use current LLM style AI.”

Thus recognising a clear difference between,

1, Current AI
2, AI Agents

With the implication that the former will not be capable of being the latter.

It’s a point I agree with and can show that it’s going to be a major problem imminently. That is,

“Current AI LLM and ML systems are incapable of acting as Future AI Agents, of any capability.”

The reasons for this are not just the lack of incremental improvements in efficiency or effectiveness of current LLM and ML systems they just are not designed or built to do it. Look at it this way

“Current delivery bots can not dance”

Essentially such bots are “a box on wheels” just like an “Ox Wagon” their design prohibits them from “back flips” and the like that human agents can do.

If we look at what is required to function as an “agent” we come up with a check list of “required” functions. Likewise we have a capability checklist already for LLM and ML systems. If we were to compare the “required for agent” and “capable with LLM&ML” checklists a couple of obvious questions arise,

1, What would the gap be?
2, Can LLM&ML expand to fill it?

The current AI LLM and ML systems are already at or close to their technology minima or capability maxima depending on what way you want to view it (or as some say “they lack moat”[1]). That is they are literally “in the bag”, or cul-de-sac, so going further down it will gain you nothing of desired worth.

As Seth Godin once noted[2],

“If your job is a cul-de-sac, you have to quit or accept the fact that your career is over.”

As a manager of an “agent” you should be aware of this and either pull the agent out of the “dip” or “close them out” otherwise you end up with “a zombie on the books” eating resources[4].

It applies to “agents” in general not just human agents. So the same applies to technology development.

Some functions or jobs are mechanistic or entirely deterministic, they are essentially “known and characterised”. Thus they can be satisfactorily automated in “predictable” environments. This was known back in the 1980’s with “Robot Research”.

However robots fail when either,

1, The required response is unpredictable or unknown.
2, The environment is unpredictable or unknown.

I’m known for saying that under the Newtonian laws of physics we mostly live under,

“There is no such thing as an accident, just lack of information and or time.”

Thus those Self Driving Car “accidents” are not “acts of deity” but “failure of design”[4].

A worse fate awaits those who want AI Agents for even very simple tasks most humans can do without conscious thought.

There used to be a joke back in the early days of industrial robotics last century in the UK due in part to a steeple jack called Fred Dibnah and an off hand comment he allegedly made about assistants being of “No bl**dy use” because they could not make a cup of tea.

The fact is any apprentice in a factory in the UK can make a cup of tea –even today– just by being asked in one of hundreds if not more ways. You would be hard pressed to find any robot anywhere that could do the same.

Thus there is also an “Expectation Gap” between the capabilities of a human agent and a Robot / AI agent. It’s something that few will be able to cross because sadly most are now not as cognitively adept as people were just a generation or so ago.

It’s something that shows up more and more frequently in the work place and why “bringing back manufacturing jobs” to the US is not going to happen even in three or four generations…

As for AI agents to use them you have to understand how they work at rather more than just the “user interface level”. If you doubt this try getting someone “in accounts” or similar that uses computers as their job function to make a simple design from scratch on a 3D printer using CAD/CAM software… You will get a tiny feel of what is going to happen over the next decade or so.

[1] The Moat in question is a more general concept of the “Economic Moat” of Warren Buffet,

https://pictureperfectportfolios.com/warren-buffetts-views-on-economic-moats-analysis/

In essence it’s a form of a combined notion of a “Perimeter Defence” and “Defence in depth”. In essence it buys you time and knowledge by which you can fend off attackers.

Unfortunately for current AI LLM and ML system developers the usual AI issue of “illusory moat” crops up with each new tiny step hence the AI bubbles every few years.

For instance Sam Altman and his investors including Microsoft thought that they had significant or unassailable “moat” due to the way model building needed lots of resources that nobody else had access to. Then at the begining of the year along came DeepSeek ripped off his crown and stomped it into the dirt and revealing it was not just broken but made of gilded brass and glass not gold and diamonds. Which caused the markets to barf and OpenAI to try hiding behind the “out house hedge” of Executive Orders and Chinia Ire that fills the void.

https://www.ctol.digital/news/openai-rethinks-china-strategy-as-deepseek-disrupts-ai/

The simple fact is such disruption was quite predictable and hiding behind EO’s or even US Legislation will actually not help. And may cause significant collateral damage such as the EU legislating Google
Microsoft, Meta, and similar AI and it’s use illegal due to various existing legislation to do with personal and private information.

[2] The quote is from Seth’s little book “The dip” and is about what we now call “Strategic Quitting” or “getting out ahead of the game or gun”. In effect it requires you recognising you are in a trap not a dip, and you need to get out whilst you have the ability to do so[2].

[3] Seth indicates there is a dip/trap difference with,

“You can know before you start whether or not you have the resources and the will to get to the end. It’s pretty easy to determine whether something is a Cul-de-Sac or a Dip.”

However… The three unanswered problems of that are how can,

3.1, “You know before you start”?
3.2, If “you have the resources”?
3.3, It’s possible “to get to the end”?

That is achieve an objective before you exhaust your reserves, options or both. Seth like many “Executive gurus” does not answer those questions, because the questions are them selves an unsolvable trap (thus makes Seth’s comments moot outside of a narrow and oft meaningless viewpoint).

The answer to 3.1 is “you can only know” if what you are thinking of doing has “been tried many times before and written up” such that you have adequate examples to make a valid hypothesis. In engineering it’s what project “History Files” are all about. They are the most essential part of project documentation as they “carry forward” what has been learnt or can be learnt from the project (see the “scientific process” as to why).

The answer to 3.2 is the same as 3.1. that is “you can only know” the resources required if you know what is required to finish the project. The slight difference is “process improvements” generally mean less resources for a given function over time.

The answer to 3.3 is what the famous Church Turing Thesis of the “halting problem” is all about. If a function has no known or predictable end point. Then it’s not possible to answer any of the questions.

The catch of course was highlighted a year or two before by Kurt Gödel though nobody really noticed at first. In essence Gödel pointed out that the really big problem was that the system/agent did not have the ability to actually recognise traps and resolve them. Gödel showed for non simple well found logics this was not possible…

Hence the all to frequent application of the human statement of,

“Take a blind leap of faith.”

Which most would not buy a book to be told.

[4] Sometimes as a manager it’s desirable to have some one “held in a dip or potential trap”. Most “agents” don’t realise that this is about 1/3rd to 2/3rds of their job/project or work function anyway.

We call it “capacity planning” but it can also be “make work”. In both cases it’s purpose is oft too difficult for many to understand without a lot of background (something DOGiE does not have). To do so you first have to have a very good understanding of the total “process chain” of which the “value added” and “supply” chains form very small parts of. The best way to understand it is to study “natural systems” and their resilience to threats such as supply shortages and attacks by other entities. In human terms we talk about,

“Putting by for a rainy day”
“Building slack into the system”
“Keeping a weather eye”
“Knowing which way the wind is blowing”

In essence they are an integration and prediction process to deal with peeks and troughs in the process chain, such that what appears chaotic can be smoothed out so “trends” can be spotted and used for “planning”.

The reason that Spain and Portugal had a major “power outage” a few days ago will eventually be found to be due to a lack of “slack” in the system that tripped and cascaded. Most probably caused by the fact that most grid connected “renewable energy” systems are designed not to have “fly wheel reserve” equivalent (the reason for this oddly enough is for both reliability and safety, but that’s a long story).