Schneier on Security

Clive Robinson • September 12, 2024 11:59 AM

@ ALL,

Re : Reward is not a good selection of fitness of function nor are the heuristics behind it.

As I’ve mentioned in the distant past of the 1980’s I had involvement with the “AI of the Time” which was “Expert Systems” and “Fuzzy Logic” that I combined and used for “Near real time physical modeling”. Which today would be done by DSP systems and Matched Filters. Which in Current AI terms would be in effect part of “Machine Learning”(ML).

In effect AI “Large Language Models” use “statistically weighted models” that are in effect “Filter Banks” of “Matched Filters” in multidimensional vector space based on quite simple networks of what are DSP “MAD Networks”. That is the single instruction “MAD” stands for “Multiply and ADd” where by each input signal is multiplied by a “weight” and the result is added to a total of all inputs. This total is then subject to a “non-linear” output transform (think either very simple like all negative values are set to zero). Or a simple “lookup table” to do line shaping such as turning a “saw-tooth” linear output into a piecewise approximation of a sine wave or other complex function such as a “Sigmoid function”. Something that has likewise been done in “audio processors” to get “soft limiting” or as in phone networks “echo cancellation” and also “bit compression” going back before the 1980’s where a 12bit linear ADC output is compressed to 8bit A-Law or u-Law.

Thus “The magic sauce” is the MAD input multipliers or “weights” and what result the non linear transform on the output modulates it with. As a matter of fact, current LLM “Digital Neural Networks” actually do things the wrong way around. That is they should do the nonlinear transform on each input and linear scaling on the output.

But also “going back to the future” of the 1980’s, the “Expert Systems” were basically just “Rule” or “Decision” trees whereby a question would be asked and the answer used to go down one branch or another of the tree untill you reached a leaf node. Any one who has tried to build one of these knows that “in theory” they are excellent, but “in practice” very difficult when used on real world problems. I could go through a long list of the failings of “Expert Systems” but many will have a feeling for them if they have ever tried to build a moderately complex software from specification by “flow charting it out”.

One area where problems can arise is short cuts and rules of thumb that get the fancy title of “Heuristics”. They have fancy definitions and get called things like “aids to learning” and similar positive spin statements… However those that have tried to use them in automated systems such as AI systems would probably nod in agreement to,

“What Are Heuristics?
These mental shortcuts lead to fast decisions—and biased thinking”

Used as the title and intro to,

https://www.verywellmind.com/what-is-a-heuristic-2795235

Where they go on to further say,

“However, heuristics have both benefits and drawbacks. These strategies can be handy in many situations but can also lead to cognitive biases.”

(Reading the rest of the article would help give perspective to many where things that give the “WTF” startle response appear inexplicable).

The thing is “in humans” we can see where heuristics start going wrong and often “head up a cusp” or behave in an asymptotic way. And to certain extent so can AI systems.

However in AI ML they can “go down the rabbit hole” in oh so many other ways, especially when humans don’t think it through…

Oddly this has been known since before the uptick in Expert Systems and Fuzzy logic in the 1980’s.

I was reminded of it just yesterday when watching a YouTube of a 1986 talk given by Nobel Physicist Richard Feynman,

https://m.youtube.com/watch?v=EKWGGDXe5MA

Have a scoot along to ah hour and nine mins in and listen to the answer to the ladies question about relations and listen to the story Richard relates about the Naval War Game and importantly a little later “Heuristic 693”.

From this you will realise why RLHF is problematic at best and likely to go wrong in ways that will not be classified as “Hallucinations” but will get a comment such as “What is this thing on?” or worse. Maybe “Mad Machinations” will be adopted as a “term of art” in AI ML 😉