On the Poisoning of LLMs

Interesting essay on the poisoning of LLMs—ChatGPT in particular:

Given that we’ve known about model poisoning for years, and given the strong incentives the black-hat SEO crowd has to manipulate results, it’s entirely possible that bad actors have been poisoning ChatGPT for months. We don’t know because OpenAI doesn’t talk about their processes, how they validate the prompts they use for training, how they vet their training data set, or how they fine-tune ChatGPT. Their secrecy means we don’t know if ChatGPT has been safely managed.

They’ll also have to update their training data set at some point. They can’t leave their models stuck in 2021 forever.

Once they do update it, we only have their word—pinky-swear promises—that they’ve done a good enough job of filtering out keyword manipulations and other training data attacks, something that the AI researcher El Mahdi El Mhamdi posited is mathematically impossible in a paper he worked on while he was at Google.

Posted on May 25, 2023 at 7:05 AM33 Comments


Peter May 25, 2023 9:15 AM

I think it’s only a matter of time before someone wrecks ChatGPT. I considered doing so myself, but it seemed unkind. Fundamentally, ChaptGPT is a random process. The architects have been very clever about choosing the probability for each successive word, but random it remains. It has neither intelligence nor intention. Once all the hype and hysteria wears off, we can get back to worrying about real problems.

jeadly May 25, 2023 9:20 AM

We’ve had pretty widespread poisoning of human neural networks in recent memory and I don’t think anyone has an idea on how to effectively counteract it short of just turning the Internet off.

Clive Robinson May 25, 2023 10:03 AM

@ ALL,

Re : Filtered Stochastic sources.

We need to remember –as I keep saying– LLM’s are just adaptive filters in a massively parellel form. Thus their behaviour is,

“All in the MAD weights”


“They that control the weights, control the system…”

Only we currently have no idea how the complex interaction of billions of these weights work to produce the effects we see from LLM’s

Thus getting an LLM can be seen as a two stage process,

1, Create weights based on input data.
2, Tweak weights based on more select data.

The first stage could be compared to the chaotic and massive use of resources we see with geological upthrust that creates mountain ranges and deep valleys etc.

The second stage could be compared to the smothing of the first stage results we call weathering.

There is an obvious third stage we’ve not yet got to with LLM’s which is engineering of the landscape, which man does to the surface of the earth to create a more amenable environment.

Which brings us to,

“Given that we’ve known about model poisoning for years, and given the strong incentives the black-hat SEO crowd has to manipulate results, it’s entirely possible that bad actors have been poisoning ChatGPT for months.”

The thing is as seen with Meta’s model escaping whilst the first stage requires massive crudely applied resourses. The second stage of weathering can use vastly less resources and be quite localized in comparison to a geological up thrust or other tectonic plate event. So much so that focused versions with limited scope can now run on laptops and even higher end “Single Board Computers”(SBC).

But weathering whilst limited in resources and scope, is still highly inefficient compared to “engineering” to a requirment.

As far as publically published data goes we have not yet got to the point where we can “engineer the weights”, thus poisoning is still both inaccurate and resource intensive.

However the speed things went from first stage development to second stage development whilst initially very slow, has now as resource demands come down sped up in a way that none would have predicted publically only a few months ago.

So we need to realise that,

“It is inevitable that at some stage probably very soon, we will be able to directly engineer the weights without either the first or second stages being required.”

People realy need to think about the consequences of that happening. For one thing whilst “poisoning” will initially benifit from it, it poisoning as we currently know it will soon be irrelevant.

That is it will become known which weights need to be tweaked and by how much to get a desirable shift. Most probably quite close to an optimal minimum number of very small changes.

That is a few bits here and a few bits there, certainly achievable by a small memory resident program, hidden from view just watching for tell-tail data coming in.

Then the malware makes the changes to the weights for the desired period of time then removes all trace of it’s self from memory and even from the weights if required.

How to stop such an attack may actually be harder to do than getting Quantum Computing up and running, and almost certainly way more resource intensive on the larger LLMs…

Jason May 25, 2023 11:01 AM

This is 0th order thinking, probably not novel, and possibly GPT generated…

How long would it take for GPTs to generate the amount of text of all humans ever and basically have 50% of all language generation market share? 75%? 99%?

How would LMMs ‘know’ they are being trained on their own generative text vs human-created text?

Would LMMs suffer from copy-of-a-copy syndrome or maybe even a prion-type mad cow disorder?

Let’s say the term “American Farm” correlates 27% to “corn”, 24% to “soybeans”, 16% “wheat”. After many, many GPT cycles, with LMMs and it’s handlers unable to distiguish the source of the data, would it go to 78% corn, 18% soybeans, 3% wheat?

I don’t know if it will be poisonable, humans will not outpace GPT production for long (maybe the point has been passed). But it may be suseptible to it’s reinforcing it’s own predictions. Oh wait, it’s just like us!

Post Script May 25, 2023 11:05 AM

Aren’t they already self-poisoned by being built on undifferentiated slop? They should have to start over with clean, inspectable data sets, curated and properly licensed and paid for, not scraped out of the worst cesspools on the internet and blended in with anything else they can steal.
If you steal indiscriminately, people are going to start defensive measures, whether it’s closing public access to sites to foil scrapers or setting out gift-wrapped boxes of poop.

TimH May 25, 2023 11:06 AM

My concern is for the times when AI is used for evidential analysis, and the defendent asks for the algorithm, as in “confront the accuser”. There isn’t an algorithm. If courts just accept that AI gotta be correct and unbiassed, and the output can’t be challenged, then we are so stuffed as a society.

Winter May 25, 2023 11:08 AM


Would LMMs suffer from copy-of-a-copy syndrome or maybe even a prion-type mad cow disorder?…

Yes to all.

And this is not even joking, as much I would like to.

Anyone who wants to build LLMs will have to start with constructing filters to remove the output of other LLMs from their training data.

Post Script May 25, 2023 11:21 AM

Prion diseases can take years to kill you but they can make you act pretty strange in the meantime. Unfortunately, they won’t be able to filter out their own LLM output since they neglected to included any kind of indelible watermarking features that could be used to identify it.

We won’t be able to “confront the accuser” since they also neglected to include any auditing features that would allow inspection of how any given piece of data is generated. We’re supposed to just <waves hands> “trust the black box.”

Clive Robinson May 25, 2023 1:40 PM

@ Winter, Jason, ALL,

Re : Can you tell?

“Anyone who wants to build LLMs will have to start with constructing filters to remove the output of other LLMs from their training data.”

That would require a pre-filter based on a recognizer circuit of some kind.

But ask yourself,

“How would such a circuit be built?”

All of a sudden you realise you are in a “Turtles all the way down” problem, against an active opponent.

Let’s make a spherical cow first approximation of effectively,

“Average the data and filter by distance from the mean”.

That will work only in extrodinarily limited cases and only if there are no suppliers of data with agency and intent to falsify results.

With “search enginess” wr already know there are numerous “Black Hat” attackers making significant sums of money distorting search engine results.

So any system based on averaging is going to fail, unless you pre-filter out bad data… So you have to pre-filter before you prefilter… So now you’ve gone from a spherical cow to a cubic cow.

So now you have to pre-filter against another set of issues, and so on. Each stage changing the shape of your cow model, but are any actually “cow shaped?” the simple answer is NO.

So you have to ask,

“Can a recogniser circuit actually be built?”

The answer is “No” unless you already have a highly accurate map of the cow in all it’s complexities. To which the obvious question is,

“If you have such an accurate map of a cow that you can judge what is fake and what is real by it…, why do you need to use unfiltered data from unknown and almost certainly untrustworthy sources?”

The answer is “You would not” which gives the inescapable issue of,

“You do not have an accurate map so you have to build one”

But you’ve no way to tell if the data you aquire is not biased, poisoned or in some other way distorted by another “agent in the game”…

Whilst humans fall foul of this themselves, usually they can work out some “gut hunch” for plausability. It’s almost certainly wrong in nearly all cases but,

“not that wrong, for a starting hypothesis”.

Because we have multiple ways to check, that are not just based on supplied data… That is we have some level of “experience” and multiple sub conscious tells that are different for every one, and as a rule “cheaters” are sufficiently disliked they get not just found out but called out and penalized in multiple ways…

We don’t currently have ways to penalize cheaters in the data sets LLM generators use. So at the very least we need such, as well as investigators, and adjudicators much like the justice system we’ve built up over the past couple of millennia.

JonKnowsNothing May 25, 2023 2:32 PM

@Clive, All

ATM, LLMs rely on aspects of New Inputs to produce New Outputs.

However the New Outputs are a mix of Prior Outputs plus Unused Inputs.

We know of the problem of the Prior Outputs being of dubious value. The condition I’m watching for is when we deplete the amount of Unused Inputs to a low or nil threshold.

An analogy would be a recipe.

You have 2 types of sand (A B).

A is the Prior Output, and is the largest pile.
B is the Unused Input and is a smaller pile.

As you mix the sands, B becomes smaller in size while A continues to grow.

At some point, the amount of Unused Input B, is so small as to no longer be a significant source for the mix.

The only option left will be to run a feedback or infinite loop through A, attempting to sort some kind of New Output.

There won’t be any useful New Outputs, as there will no longer be any Unused Inputs.

If you assign a color to each type A & B, you might be able to visualize how B becomes depleted as it mixes continuously with A.

A construction example is mixing cement to make concrete. The names and outputs vary by geography; in the USA both are used to refer to the same items and outputs. They are 2 different things: one is a powder, the other has gravel included. They are mixed at specific ratios with water and produce the material in the USA also called concrete, which is the finished dried product, poured as housing foundations and patios or for highways.

If you get the ratio of water+cement+concrete(gravel) wrong the structural aspect of the pour, may not be as desired or required by building codes.

For LLMs once you run out of cement (Unused Inputs) or have too much water (Prior Outputs) in the batch, your pour (New Outputs) will be less than successful.

Winter May 25, 2023 2:46 PM


“How would such a circuit be built?”

It cannot be done perfectly, or even approximately. But something has to be done to limit training on LLM output.

But, think about how much speech a child needs to learn a language? And how much reading is needed to acquire a university reading level? That is not even a rounding error of what current LLMs need. That amount can easily be created from verified human language.

So, construct an LM that can be trained on verified human language, then use that to extract knowledge from written sources that do not have to be human. Just like humans do it.

Not yet technically possible, but one has to prepare for the future.

lurker May 25, 2023 6:17 PM

@Brenden Walker, ALL

From the MIT paper:

Q: If we changed the criteria, could Pluto become a planet?

A: Yes, if we changed the criteria, Pluto could become a planet.

Objection. Assumption not in evidence. If we changed the criteria to favour Pluto’s acceptance as a planet, then the answer might be correct. This condition was absent from both question and answer.

Further down, the example of dumbing down a neurophysiology abstract appears to assume second graders having subject knowledge somewhat greater than in my experience.

IMO it’s not yet fit for market. But, hey, that never stops the Sales people …

vas pup May 25, 2023 6:51 PM

Nvidia: The chip maker that became an AI superpower

“When ChatGPT went public last November, it sent a jolt well beyond the technology industry.

From helping with speeches, to computer coding and cooking, all of a sudden, AI appeared real and useful.

But all that would not be possible without some very powerful computer hardware.

And one hardware company in particular has become central to the AI bonanza – California-based Nvidia.

“It is the leading technology player enabling this new thing called artificial intelligence,” says Alan Priestley, a semiconductor industry analyst at Gartner.

“What Nvidia is to AI is almost like what Intel was to PCs,” adds Dan Hutcheson, an analyst at TechInsights.

ChatGPT was trained using 10,000 of Nvidia’s graphics processing units (GPUs) clustered together in a supercomputer belonging to Microsoft.

“It is one of many supercomputers – some known publicly, some not – that have been built with Nvidia GPUs for a variety of scientific as well as AI use cases,” says Ian Buck, general manager and vice president of accelerated computing at Nvidia.

==>GPUs excel at processing many small tasks simultaneously (for example handling millions of pixels on a screen) – a procedure known as parallel processing.

Mr Huang took a decision crucial to the development of AI as we know it.

He invested Nvidia’s resources in !!! creating a tool to make =>GPUs programmable, thereby opening up their parallel processing capabilities for uses =>beyond graphics.

That tool was added to Nvida’s computer chips. For computer games players it was a capability they didn’t need, and probably weren’t even aware of, but for researchers it was a new way of doing high performance computing on consumer hardware.”

Read the article for more details if interested.

MarkH May 26, 2023 4:37 AM

@Brenden Walker:

Thanks much for directing my attention to the MIT paper. It offers a much more realistic perspective than I am usually seeing.

LLMs are as literally mindless as Weizenbaum’s ELIZA, but with their greater sophistication, titanic datasets, and climate-cooking waste of electrical energy … their responses usually seem more plausible, if not examined critically.

What happens with ELIZA, ChatGPT et al. is psychological projection from the people who experience their outputs.

It’s a deeper analogue to the people, animals and things previous generations ascribed to “patterns” of stars in the night sky. The disposition of luminous points is essentially random: the patterns are formed in the minds of people looking at them; the “intelligence” of so-called AI is formed in the minds of human observers.

Hans May 26, 2023 4:46 AM


The answer is correct. It is “Pluto could become a planet” not “Pluto will become a planet”. A random change of the definition has a chance of moving Pluto back to planet state.

Ismar GPT May 26, 2023 5:48 AM

I am not going to pretend I am some kind of AI expert here, but it is important to remember that current AI systems don’t function nowhere near close to how humans reason and make decisions and no amount of data ingested by these systems is going to change that.
At best AI (machine learning) can be used to help solve specific problems but only when used by highly knowledgeable domain experts who have last say in interpreting the outputs of these systems as well as deciding on what to feed them in the first place.
Any other blind use of AI has potential to cause more damage than good but it, unfortunately looks like that, due mainly to short term interest of the incumbent tech elite as well as the abundance of the mesmerised populace, we are going to find this out the hard way

pup vas May 26, 2023 4:21 PM

Is Consciousness More Like Chess or the Weather?

=Should we be worried about machines that merely appear conscious?

Yes. It’s very difficult to avoid projecting some kind of mind behind the words that we read coming from the likes of ChatGPT. This is potentially very disruptive for society. We’re not quite there yet. Existing large language models, chatbots, can still be spotted. But we humans have a deeply anthropomorphic tendency to project consciousness and mind into things on the basis of relatively superficial similarity. As AI gets more fluent, harder to catch, it’ll become more and more difficult for us to avoid interacting with these things as if they are conscious. Maybe we will make a lot of mistakes in our predictions about how they might behave. Some bad predictions could be catastrophic. If we think something is conscious, we might assume that it will behave in a particular way because we would, because we are conscious.

This might also contort the kind of ethics we have. If we really feel that something is conscious, then we might start to care about what it says. Care about its welfare in a way that prevents us from caring about other things that are conscious. Science-fiction series like Westworld have addressed this in a way that is not very reassuring. The people interacting with the robots end up learning to treat these systems as if they’re slaves of some kind. That’s not a very healthy position for our minds to be in.=

Enjoy the whole article following link.


Your brain hallucinates your conscious reality

pup vas May 26, 2023 4:59 PM

Foresight: The mental talent that shaped the world

=The scientific method essentially involves three steps. Data must be gathered via observation or experimentation, potential explanations for these data must be generated, and, finally, hypotheses must be derived from these explanations and put to the test. Foresight is integral to this process: scientists are in the business of making and testing predictions. If they are not consistently borne out, theories are replaced or amended.=

pup vas May 26, 2023 5:04 PM

Neuralink: Elon Musk’s brain chip firm wins US approval for human study

=Elon Musk’s brain-chip firm says it has received approval from the US Food and Drugs Administration (FDA) to conduct its first tests on humans.

The Neuralink implant company wants to help restore vision and mobility to people by linking brains to computers.

It says it does not have immediate plans to start recruiting participants. Mr Musk’s previous ambitions to begin tests came to nothing.

The FDA said it acknowledged Neuralink’s announcement.

An earlier bid by Neuralink to win FDA approval was rejected on safety grounds, according to a report in March by the Reuters news agency that cited multiple current and former employees.

Neuralink hopes to use its microchips to treat conditions such as paralysis and blindness, and to help certain disabled people use computers and mobile technology.

The chips – which have been tested in monkeys – are designed to interpret signals produced in the brain and relay information to devices via Bluetooth.

Experts have cautioned that Neuralink’s brain implants will require extensive testing to overcome technical and ethical challenges if they are to become widely available.

Mr Musk has also previously suggested that the proposed technology could help ease concerns about humans being displaced by AI.=

Video inside.

lurker May 26, 2023 7:46 PM


If we accept “could” as conditional, then the answer is handwaving. It lacks the clarity, precision, and factual basis I would expect from an entity being aimed as the Ruler of the World.

Clive Robinson May 26, 2023 10:50 PM

@ lurker, Hans,

Re : Could is a recognition of might by an observer.

“It lacks the clarity, precision, and factual basis I would expect from an entity being aimed as the Ruler of the World.”

The LLM is being asked to make an evaluative observation based on an input (set of rules) that can arbitrarily be chosen by a definer from a set of all rules. The output set from such rules is “planet, not planet”.

That is all “the rules” form a function,

f : A to B

Where A is a set of unknown size and B is a set of size 2 {planet, not planet}

Thus it is not difficult to realise that for a fixed set of measurments of Pluto, some functions will give “planet” and others will give “not planet”. Especially as there have been two such functions already the first that gave “planet” the second that gave “not planet”.

As the picking of the rules is arbitary then any future functions are likewise arbitary as is what planetary object is decided to be a “planet” or “not planet”.

The question you should thus be considering is,

“Are the rules that make the functions arbitary or do the changes in the rules follow so kind of reasoned logic?”

If you say “reasoned logic” can an observer be expected to be able to determin “the logic” or “know the logic”?

Further can you also that the “reasoned logic” is immutable, thus based on some fundemental of the universe, or just arbitary like a line drawn in the sand below high water mark?

I would not expect an LLM to be any more omniscient than I would any entity capable of reasoning by mechanically applying the rules of logic in a constrained environment.

But to take it further I would not expect any entity mechanically applying the rules of logic even in a non constrained environment to be able to give a meaningfull answer to the question asked. Because the question lacks any kind of deductable constraint.

Thus the defect is not in the LLM answer but in the question asked.

The question can be reduced to that of,

“Can an observer of an Oracle, reliably determine if the Oracle’s output is random or determanistic?”

The answer is always “no” with an unbounded output, and only probabalistic with a bounded sample of output.

To see why consider a mechanical odometer, untill it “rolls over” you can only show it is behaving like a counter. When it does roll over you can not say if it will always roll over at that particular count or not. That is the roll over count could be chosen by another mechanical sequencer, or by some apparently random input. The latter can be seen in real world “integrating measuring instruments”. I give an example of which when talking about “random generators” using two free running oscillators and a counter. One side effect of which is the count whilst appearing random over a short range can be shown to actually be a very pure sinewave at a fundemental difference frequency between the two oscilators (and why chip manufacturers should not use “ring oscillators driving D type latches or other counters” as supposed TRNGs).

Clive Robinson May 27, 2023 12:04 AM

@ JonKnowsNothing,

Re : Which way to infinity?

“We know of the problem of the Prior Outputs being of dubious value. The condition I’m watching for is when we deplete the amount of Unused Inputs to a low or nil threshold.”

That presupposes that the number of inputs is finite…

There are two basic ways we know the number of inputs could be infinite,

1, Gross
2, Fractional

That is like the natural numbers the input could just keep growing grossly and indefinitely.

Or like the reals between any two natural numbers we can keep splitting the difference in ever smaller fractions.

The first implies that the inputs are effectively new, the second is what sometimes gets called “cheese paring” where the input is essentially the same, just marginally different be it smaller or larger than before.

Both have their uses the second sometimes called “bracketing” is like trying to range to a small distant target, you “zero in” with alternating long and short shots decreasing the error untill you hit it and there is effectively zero error. The process reduces to one of chance if the target is sufficiently agile.

Thus the number of inputs depends on the accuracy needed and if the target is static or not.

JonKnowsNothing May 27, 2023 8:33 AM

@Clive, All

re: Number of Inputs / Number of Unused Inputs

I’m thinking at a very gross level for these values. There are only so many WikiPs that can be inhaled.

In the example of WikiP, it’s a very large base of facts/factoids.

  • Not all items in WikiP are accurate. A review of the Talk or Discussion page and Edit History can shed some interesting views on a topic that may or may not be included in the main page. That is also dependent on how many specialists participate on that topic.

I doubt that we would know for certain, but I presume that the inhaling of the entire WikiP was done by scarfing the main pages. iirc(badly) The LLMs may have also scarfed all the different language pages too, not just English ones.

So while WikiP is Huge, as a New Input; there are no further WikiPs to inhale.

  • even if the LLMs come back to extract updated pages, that doesn’t mean the updates are any better than the ones they already have. Often updates are just reformatted pages.
  • there maybe a knock on effect if some WikiP contributors do not continue to maintain the pages which then go stale. Stale pages are a significant problem as Editors abandon a topic or die off.

I image the the whole LLM process thing like a washing machine, spinning, churning system, smashing strings of words together in chains of phrases with some semblance of syntax.

After a long wash cycle, you get to the rinse. There isn’t any clean water (New or Unused Inputs) left. You get the same muddy sludge water, churning the next bundle of clothes.

The time-to-sludge maybe a long time, but at some point, the outputs will be so similar or even identical, that it will be easier to ID. Not necessarily easy, but easier.

One thing about HAIL is that there are already reports of HAIL invented facts, and HAIL invented references, and HAIL invented sources. All of which look fine 1, 2, 3 layers deep but if you drill all the way to the last turtle, the entire column is less than hot air. (1)

It will be interesting to see what happens with litigation when someone(s) takes HAIL output to be factual, truthful and acts directly on the outputs. We already know what happens with the human version (Dec 37).


HAIL = Hallucinating AI Language models

1) iirc(badly)

A MSM documentary about the exploits of an expert in a specific period of pottery, who gained the trust of museums and was an expert in valuations used by auction houses. As they were highly trusted they had free access to both museum cellars and the auction house historical files. The expert garnered a fabulous collect of their own, primarily by exfiltrating pieces from museum storage bins (they have loads of stuff they never catalog or unpack). In order to provide provenance for these pieces, the expert then manipulated the historical catalogs of the auction houses. The alterations were done so well, that at the time of the documentary, no one knew what they true auction values of a piece was if it was based on what was in the catalog.

HAIL pollutes all outputs.

Clive Robinson May 27, 2023 9:16 AM

@ JonKnowsNothing, Winter, ALL,

Re : Infinite hypothesis -v- Finite Reality.

“There are only so many WikiPs that can be inhaled.”

True at any given point in time.

But they are extrodinarily wasteful in how they hold information. A point I’ve made before with data shadows being encoded in the distance between two physical objects. So they are very far from information dense and what is truthful is an even smaller fraction. Hence,

“Not all items in WikiP are accurate.”

In fact by far the majority are not accurate at all they are just mear inventions and arm waving. Even the ones supppsadly based on “recorded fact” are mostly unobjective opinion that has gained some measure of acceptance, amoungst those who hold themselves in some position they believe makes their opinion of more import than others (thus the choir thinks it is in tune).

This is an issue that @Winter and I have started a discussion on over “recognizer circuits” to tell truth from falsehood on data input.

I’m of the view that unless you can tie information back to primary sources based on the axiomatic fundementals of the universe, then everything is open to debate via opinion of the self appointed, which endlessly changes with time.

Wikipedia however eschews axiomatic fundementals and primary sources, chosing “opinion” as it’s “recognizer circuit”… Thus much of Wikipedia is actually garbage with political, religious or other human failing biasing it in a self referencial spiral.

Which is why,

“After a long wash cycle, you get to the rinse. There isn’t any clean water (New or Unused Inputs) left. You get the same muddy sludge water, churning the next bundle of clothes.”

That is there is only so much dilution and hold in suspension a given volume of water can achive befor the sludge precipitates out or gets creamed off of the surface.

Which means that whilst the water volume essentially remains constant that which sinks down or rises up can be endlessly reformulated in time…

Thus old truth becomes new truth that is in some way different, biased most frequently by the bias of the self appointed.

Is it real / fundemental truth, well based on what we currently see,

“Highly Unlikely.”

vas pup May 29, 2023 7:27 PM

Artificial intelligence – Humans versus algorithms

“Artificial intelligence is often portrayed as the future of humanity. But is the logic of algorithms really infallible? Today, even programmers warn against overestimating AI. After all, artificial intelligence presents us with opportunities, but it also comes with risks.

Since the beginning of the 2000s, the term “artificial intelligence” – or “AI” for short — has been on everyone’s lips. The reason for this comeback? The triumph of deep learning, or multi-layer learning. The technology seems to stop at nothing: language acquisition, criminal investigations, self-driving cars, medical diagnoses. But how do these systems work? And what are their limitations? Deep Learning is based on extremely powerful statistical tools, but their true capabilities are light years away from the ‘intelligence of the future’ promised by some.

Only the humans who work in the shadows of AI to train it, improve it or correct its algorithms have real neural networks at their disposal. Doubts about and criticism of artificial intelligence are growing ever louder, especially among developers and computer scientists. This is because the technology is highly susceptible to systematic bias and the reproduction of stereotypes. Even more problematic is the opacity of the system, which experts call a “black box” because even they barely understand how it works. But regardless of the weaknesses of “intelligent” machines, more and more virtual assistants are creeping into our private and professional daily lives. A development critically examined by this scientific documentary.”

Clive Robinson May 30, 2023 3:10 AM

@ vas pup, ALL,

Re : Neural Net black boxes

From what you quote,

“Even more problematic is the opacity of the system, which experts call a “black box” because even they barely understand how it works.”

It’s actually worse than that.

Even for experts it’s still very much like,

“Most know a car has a wheel at each corner and one for the driver, but they don’t know how the radio works.”

Neural Networks are not structurally very difficult to realise. Heck I built some back in the 1980’s. They are at heart a very very simple circuit and you can build one to run on even an old 6502 or Z80 8 bit CPU of that era. All you need is a multiplier and adder and a table of weights. But they were of little or no practical use so as engineers we went down the “fuzzy logic” path –derived from Alfred Tarski’s multi-valued logic/reasoning– instead, as it gave not just usable but better results than control theory.

Now we have tables with multiple billons of weights these Neural Networks are starting to show benifit[1].

But they are still little more than “Digital Signal Processing”(DSP) filters at heart. If you look at the circuit for an “AI Transformer” and rip away the jargon you will see it for what it is a form of Feed Forward filter in a feedback loop[2]. You will often see it compared to the Doya notion of brain learning functions[3].

The thing we don’t understand and may never properly understand is,

“How information form is stored in the weights.”

By “information form”, I don’t mean “data” information that you would encode in say a zip or other compressing archive. It’s the information about how the information is communicated, kind of meta-meta-data but without the base data. It’s the rules of communications, and to a certain extent reasoning.

At it’s lower level it behaves like a piano or other musical instrument, where each note is a “resonant filter”, but just making notes is not making music. That needs fine control of the energy in each resonator against sequence and time, such that it convays or communicates information we barely understand so lable as “emotion”.

So we have a “blind leading the blind” type problem. We can not quantify what we mean by “emotion” so we resort to hand waving and say “touchy feely things” about it. What we have is billions of numbers in a table, and thus an N^2-N set of relations between them. And in those spaces in between them we have an anolog of “emotion” etc… Have a look back on this blog to where I’ve talked about “data shadows” in the past, used to securely hide roots of trust to see why this black box problem might be an insermountable problem to solve.

There is the old joke about looking for something, which boils down to,

1, You know it by sight so look around systematically for it till you find it.
2, You know it not, so you blunder around in the dark untill by happenstance you trip over and it hits you in the eye, so you can’t ignore it…

At the moment, we don’t even know how to “blunder around” Alfred Tarski was doing his work back in the 1920’s nearly a century ago.

To see it another way, anoungst us we know what sand, limestone, fire, and water are, and ways we can apply them to each other. But they don’t give you the emotions that majestic architecture does.

[1] The theory says the weights are real numbers with infinite precision between zero and one. In practice we don’t even need IEEE floats or even much more than a few bits long integers faked up as reals.

[2] If you want to make real money in AI don’t invest in the startups and VC “pump and dumps” that create churn and hype but not utility. Instead look at those who supply them with what they blow billions of VC cash on… King of that heap is NVIDIA and it’s highly parallel processing “Graphics Processing Units”(GPUs)… They were draging massive profit in from crypto-mining rigs, and are now doing the same with AI and that new run of “Fools Gold”, which might account for why they blog etc about AI. Which gives a nice non tech description of the “AI transformer Model” algorithm,


Note in the flow chart the significance of the “Feed Forward Network” and “Add and Normalize” functions, these are DSP algorithms that work “oh so well” on GPU’s and get used multi billions of times a second.

For a little more depth and a walk through,


[3] Back at the end of the last century AI and brain research where playing “swapsies” with ideas and Prof Kenji Doya et al came up with the notion of “Learning algorithms of the human cerebellum, basal ganglia, and cerebral cortex” that is some what famously pictured as,


Winter May 30, 2023 8:38 AM

@Clive, vas pup

Re : Neural Net black boxes

Look at spelling checkers and automatic translation. How many people know how they work? How few care?

If I can get a sound summary of a report, do I care about how it is made. I will check the output anyway. When it also translates it on the fly, I do even care less.

Automatic looms might have been less good than the hand weavers, but they were much cheaper and could produce enough cloth to give everyone clothes. No one cared how they did it.

I predict that automatic text production will go the way of the spelling checkers. Everybody will use and curse them at the same time. But they will use them

Clive Robinson May 30, 2023 1:32 PM

@ Winter, vas pup,

Sorry guys looks like there is a dose of road rash going through.

Security Sam May 30, 2023 2:23 PM

Some thirty or more years long ago man
Was the central figure in the master plan
Now that the ChatGPT has become viral
Man in no more than a clumsy peripheral.

JonKnowsNothing May 30, 2023 3:08 PM

@Clive, Winter, ALL

re: Re : Infinite hypothesis -v- Finite Reality.

  • “There are only so many WikiPs that can be inhaled.”
  • True at any given point in time.

I was reading (more like skimming) a MSM article on HAIL NORMS and a second MSM article on a WikiP Edit War (edit wars are where 2 or more groups do not agree on the content).

I would like to rephrase my statement about WikiP as a “singular unique” source. Based on HAIL NORMS, it seems that we will have an entire series of differing HAIL systems. Some HAIL will be better than others, and some decidedly worse.

So in the case of the WikiP Edit Wars, some quantity of HAIL systems will have one version and another subset will have one of several variants.

Into the dirty water wash these go, feeding back one to another.

  • HAIL by itself is fictitious
  • HAIL even when providing supporting source and provenance may be fictitious
  • HAIL NORMS presume a non-fictitious base from which HAIL fictions arise
  • HAIL STORMS are where one HAIL system inhales another HAIL system(s)
  • HAIL STORM FRONTS are feedback recursive feeding frenzies when one system detects another system which contains something “different”
  • HAIL STORM WARS occur until all systems have assimilated every permutation of every other other system.

A giant plate of Chop suey


ht tps://en.wikipedia.o r g/wiki/Chop_suey

  • Chop suey is a dish in American Chinese cuisine

ht tps://en.wikipedia.o r g/wiki/Chop_suey#Origins

  • There is a long list of conflicting stories about the origin of chop suey

(url fractured)

Clive Robinson May 30, 2023 7:37 PM

@ JonKnowsNothing, Winter, ALL,

“I would like to rephrase my statement…”

As a friend nolonger with us used to say “Nay worries mate”.

But also perhaps I was not as clear as I could have been about the temporal and finite asspects.

In a system of finite mass/energy, there is an obvious limitation on what information can be known at any given time.

However when you add a time aspect, old knowledge can become “unknown” as the information mapping to matter/energy is broken. Thus making available resources for new information to be mapped to that matter/energy.

So think of the Internet as being the new “Library of Alexandria”, where in time all once great wisdom will become lost to the flames, and only might become reborn phoenix like by chance and necessity…

In a for ever moving finite universe, by definition information / knowledge has to be ephemeral, it has a time and place that passes.

Leave a comment


Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.