Schneier on Security

Clive Robinson • July 29, 2024 9:00 AM

@ Bruce, ALL,

Re : Demise of an arms race.

“The latest in what will be a continuing arms race”

History shows us that even in a war arms races come to an end.

With the winner generally being the side with,

1, Greatest technological capacity.
2, Largest reserves of resources.

Because of two reasons,

The first is an arms race gets expensive to some power law for each increment in capability.

The second is some technology obsoletes the technology the arms race is based on, and effectively a new arms race begins.

Perhaps not as well known as it should be but LLM AI systems in their current form have got beyond the size where the laws of physics and basic geometry allow simple scaling up, because whilst much can be done in parallel not every thing can and it takes quite a while for a signal to get from one corner of a cube to the furthest corner.

A grain of sand is considered about 1pS or picosecond in size and a one foot length 1nS or nano-second. As these are the times it takes light in freespace to travel these distances.

But there is also a density issue. Such electronics draws a European sized countries worth of electricity and in terms of power in to power out current LLMs are about as inefficient as it gets. Which means that most of that input power becomes heat. Air is not a particularly good conductor of heat and even though water is better it has a lot of disadvantages. Copper silver, gold and diamond do a much better job as conductors of heat but there are other limits on their use.

So even with optimal step down techniques to move the heat away, only about 1/10,000th of any given volume can be considered as being computing volume often less a lot less.

I could give other reasons for why current LLM AI systems are hitting the cost and resource brick walls of physics but that’s not really that interesting. We know it’s going to happen and realistically when even with significant algorithmic improvements.

What is interesting is the second way Arms Races end, that is how the current LLM systems Arms Race will be replaced with a new AI arms race. For the cycle to start all again.

Oh and at some point when things move to Carbon and potentially Iron based systems as Silicon and Aluminium reach their limits.

Clive Robinson • July 30, 2024 8:57 AM

@ Winter,

Re : Resource limitations.

“Like every arms race, the limits on this arms race will be set by resource limitations.”

But which ones will bite first?

There are limitations such as the speed of light and thermodynamics that there is no known way to beat and quite a few LLM systems have got quite close to those.

Then there is the density issues of getting energy in and heat out hence resources like power, water, and metals like copper, silver and gold are needed in high quantities.

Another issue is silicon and heat getting silicon semiconductor devices above 250C is at best problematic. Which is why some are looking at carbon and iron alloys.

But then there are “Human Failings”.

Yup call it greed or some other “sin” but it’s come to light that four organisations have been stockpiling those Nvidia H100 chips that cost upto $40,000 each.

And we are not talking little piles we are talking getting on for the better part of $100,000,000,000 tucked away out of sight.

One of those four organisations is a large Venture Capitalist. And they are apparently “renting out” their H100 systems to “Start Ups”, but not for money, but A stock equity…

But it’s interesting to note that these high end systems for LLMs show all the disadvantages that “block chain” systems did…

Clive Robinson • August 1, 2024 6:58 AM

@ Winter,

First off mixing units makes mental comparisons harder than it needs to be when trying to follow an argument.

Thus “watts” and “1pJ” are not directly comparable due to time.

But also we know that the biological neural nets function very differently as far as usage is concerned.

A current AI LLM is run effectively as a single network with all parts functioning.

Fast MRI scanning and similar show the brain functions as many almost independent networks that “ramp up and ramp down” their usage with what we believe is function (though might be something else).

Interestingly we know that whilst the output of a biological neuron might be linear a subsequent neuron receiving two or more such linear signals will not see the combination linearly.

These are not things current digital neural nets emulate at all, let alone on a spectrum of approximately to closely.

Are these paths worth investigating?

Yes because they have significant advantages in reducing amongst other things power density issues.

But aside from a few computer simulations little has been done on this, and realistically doing so would to many appear to be turning the AI clock back two or more decades.