Schneier on Security

Clive Robinson • April 11, 2023 3:30 AM

@ Tony, EvilKiru, ALL

Re : micro-payment comeback

It’s not going to happen.

Back in the 1980’s the idea of digital payments and electronic wallets got started along woth pocket gambling devices and similar. But as I’ve mentioned before I found they could not be made sufficiently secure.

Around the same time in the US phone freaking was a serious activity and people discovered how to use the PABX’s of small companies and charities to do “dial-in and dial-out” fraud[1].

But also, the CellPhone system was not secure, it was not difficult to steal a mobile phone ID and make out bound calls to far foreign places on somebody elses bill. In fact in many places criminals would set up shops where you could make calls at about half the rate of landline (POTS) charges.

With phone fraud the phone companies were extreamly tardy at sorting out the fraud that they had created by their insecure behaviours. Because they were effectively accomplices by choice, in that they got payed or took the hurt person to court and damaged their reputation or just denied them phone service etc. In the UK even the phone regulator OfComm –stuffed with revolving door industry types– were giving out the wrong legal advice quite deliberately.

Eventually in the UK the law had to be changed so that the phone companies had to swallow the cost of the fraud. So they could nolonger “externalise the risk/cost” and unsurprisingly they very quickly started making changes. Though if you ask the corporate types from back then they still say that they had to wait on technology, which was not true back then nor is it true these days as we can see with the finance industry.

On a current note, have a look at “Secure Electronic Transfer”(SET) where the Payment Card Industry tried to come up with a secure system. It failed because nobody realy wanted such a system. Their next move was “Chip-n-Spin” which was full of security faults and still is due to the in built security failings. But you listen to the Payment Card Industry and they will tell you all is wonderfull in their garden (as they are not picking up the cost of fraud just making proffit on it). The same applies to these Contactless “Near Field Ccommunications”(NFC) systems.

The game is still the same “the little guy pays” unless legislation or regulation addresses the balance. Lobbying is very inexpensive compared to the cost savings the corporates can make especially when they deflect the cost onto those who can not fight back.

Any “libertarian” type who tells you they can make it otherwise with cryptocoins or digital/E-cash is either deluded or conning you. Because as a consequence there will always be criminals working a profit out of it somewhere along the line. The thought of “free money” at nearly zero risk to them will ensure they are always “at it”. The only known way to limit or stop it, is to make the corporates clean up their act by legislation and regulation that makes them pick up the cost of such fraud…

[1] All tone dial PABX’s had the ability to be configured such that you can dial one of it’s numbers (in bound pair), and it would drop you onto an outbound pair where you got dial tone, from which you could dial to any other number. Whilst it made things like accounting easier, thus waa atractivr, it also made fraud by others easy.

Clive Robinson • April 14, 2023 6:27 AM

@ ResearcherZero, Bruce,

Re : unconventional hacking

The first paragraph you quote from the article gave me an eerie feeling of deja vu, as it closely resembles some of my own words from a short while back.

Where I asked the question about the difference between an attacker sending “a malware script” of instruction for a computer to follow and another attacker sending a human a list of instructions to follow.

In either case the model is identical except for the entity under attack one script being for “in silico” the other being “in vivo”.

These attacks whilst being “in silico” are using an attack in a “language” much closer to that of the “in vivo” script attack on humans.

Thus it brings up the vexed subject of “Neuro Linguistic Programing”(NLP). NLP is anotion first thought up in the mid 1970’s but as there is no currently credible “scientific method evidence” about NLP advocates claims, it has been sofar relegated to the pseudoscience catagory. Thus NLP has joined a number of “alternative medicine” therapies.

So could these attacks against LLM’s be seen not just as an in silico attack but a form of “NLP for AI” systems, thus become a credible research domain in it’s own right?

Thus as a technology it could be used for “good or bad” under an univolved observers moral perspective of,

1, For teaching an AI is “good”.
2, For corrupting an AI is “bad”.

But the notion of “NLP for AI” also has a flip side it can and will be used for what an uninvolved observer would most probably be regarded as “bad” via the likes of “suasion” techniques.

Another much derided idea from the 70’s was “subliminal messaging” can be seen as a form of steganography, but aimed at the subconcious rather than the concious mind. However we know that “the written word” sinks in way deeper than “non shock” images do and “instills” rather than “installs” a cognative change.

So this “bad” can be flipped to “good” if the output is used to produce more effective teaching / learning tools, that could be tuned to an individuals learning style.

Which begs the question of just how many other 1970’s era ideas about the human mind and perception might be given “new leases on life” either for in silico activities or to produce in vivo scripts?

Personally as some one who regards LLM’s as little more than a glorified “Matched filter” used as a logical progression of using a stochastic source to generate the equivalent of “human memorable” “nonsense passwords” that also arose in the 1970’s, I will be interested in how others present the notion of “NLP for AI” in future.

@ Bruce, ALL,

Perhaps an Op-Ed or “thought piece” on using LLM’s for turning XKCD style pass phrase generation “word salad” output into human “memorable sentences” might help swing the uninvolved observer view from “bad” towards “good”.

Clive Robinson • April 17, 2023 9:39 AM

@ Winter, ALL,

Re : Biasing LLM probability filters.

From what you quote,

“Aligning large language models (LLMs) with human preferences has proven to
drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT.”

Begs the,

“What could possibly go wrong?”

Question…

To which human history replies,

“We’ve burned witches and done way worse as ‘human preference’ and still do…”

LLM’s have know inate knowledge by experience and can not “sense the world” to gain it.

Every input LLMs get, is either by others selection and ordering, or some other form of internal bias or randomness. From which they just build filters to reflect the sequenced probabilities of the input.

The Roman Poet Juvinal[1] just under a couple of millenia ago asked,

“Quis custodiet ipsos custodes”

So who does “guards the guards” or as we prefer these days “watches the watchers”. Whilst it is an age old question there is no answer that surfices.

Not only is it a “turtles all the way” problem, there is not even the usuall human fudge of “Rule of Thumb”. Thus it causes problems in everything in which there is an authority or power structure in place, thus every asspect of society.

Remember what is “seen as good or bad” is always from a supposadly uninvolved thus argued as impartial observers point of view.

But both the directing mind of an act and the observer are part of a society so they all are biased in some way and that changes with time and events, sometimes both radically and rapidly.

So whilst some little old lady with dementia screams as the flames consume her something most these days would regard as abhorent, it was “public entertainment” just a century or so back.

For those who say “that could never happen now” similar still goes on in various ways as power strugles for dominance daily happen even more so today than then.

It might be instructive for those who think “human prefrence” is benign or good to look back on the “Salem witch trials” at the end of the 17th Century just a third of a millennia ago.

[1] Of Decimus Junius Juvenalis or “Juvenal” little is known even though it has been alledged he was possessed of a “Patrician’s nose”. His writings show familiarity with Scotland and Egypt, though if this was first hand or not is not known. What he is remembered for is his surviving five books of satirical poems, that have been “classically studed” and later, much later quotes taken from.

Clive Robinson • April 18, 2023 5:31 AM

@ Winter, lurker, ALL,

Re : Only Obvious when seen.

“[B]iased decisions are a human trait. LLMs are just putting them in the limelight”

It is a point that is obvious when seen, but first people have to see it and those making money or using it to avoid responsability, are trying quite hard to stop it being seen.

After all LLM’s are only quite obviously,

1, Creations of Man.
2, Do not have the necessary safety features.
3, Used in ways where serious legaly defined harms have repeatedly happened to people.

Thus they are a technology that in US prosecutor behaviour certainly falls under the US legal definitions of WMD.

As you ask,

“But would you prefer a system these biases are hidden in proprietary code and data”

Absolutly not, such would only encorage significant abuse bye “those making money or using it to avoid responsability”

I’d actually like to see LLM’s be got rid off, as they have at a high level broadly the same failings as cryptocurrency but actually a lot worse. That is so far their use for “social bad” is much greater than any use for “social good” so far claimed and to be honest I can not see that changing[1].

However as you note,

“Just as gunpowder and automated looms could not be suppressed, LLMs are here to stay.”

Those atleast did and still do have reasonable arguments for “social good” unlike LLMs.

So at the very least LLMs and similar need to be a heavily legislative controled and regulated industry with significant but open Oversight (as legislation was once supposed to be long prior to the likes of the PATRIOT Act, RIP Act, and similar.

[1] Not much is currently said, is about the fact that,

All LLM’s are actually very powerful ‘Surveillance Tools’ capable of identifing and following even people taking significant precautions to prevent it”.

Thus the plans by ethically questionable companies with morally dubious connections to the likes of US “Guard Labour” and “Inteligence Community” Agencies to put LLMs and simillar into ‘search engines’ and ‘Cloud based productivity’ systems should be of very great concern to all.

Clive Robinson • April 18, 2023 8:21 AM

@ Bruce, ALL,

Re : Further thoughts to consider.

“But while it’s an easy experiment to run, it misses the real risk of large language models (LLMs) writing scam emails.”

As noted earlier with LLM’s and similar AI currently “the social bad” far outweighs “the social good”. And with a little thought people will realise “scam emails” will be the least of our problems with “the social bad” of LLMs and similar AI tech.

But argument will be made by proponents that the above view is,

1, Chicken little scare stories,
2, It’s early days, and similar happens with all technologies only to change around later.

Both at the same time along with many similar. The problem is such proponents arguments are effectively untrue even though they might have tiny pieces of truth to them, used as a smoke screen.

Unfortunately “the social bad” can and will outweigh any potential “social good” and this is not a “gut feeling”, “hunch”, or “hinky fealing” it’s based on reasonable assumptions about technology and certain types of people.

So I’ll give a simplified 20,000ft explanation, so any one who cares to think about it can consider it no matter what their background.

As I’ve indicated in the past there is a “spectrum” between “Fully determanistic”(Fdet) and “Truely random”(Tran). You can draw it out as a line between those two points,

Fdet|—+–+—–+—+—|Tran

Along which there are identifiable zones of transition that you can lable in various ways but as a rule of thumb has some form of increasing complexity or sensitivity from Fdet. So you will find “non linear”, “non continuous”, “chaos”, “Psudorandom”, “faux noise” etc along that line depending on how you want to define them, but that labeling is not realy relevent to this argument.

Because it can also be seen that running along that line is “security” and “privacy” to join “complexity” and “sensitivity” but inversely so that the highest level of security not unexpectedly is at the minimal complexity Fdet end of the line.

Now lets consider something that is orthagonal / tangential to the line and call it “surveillance”.

Currently we are transitioning from tangible physical world surveillance to intangible information world surveillance.

Three key characteristics of implementing surveillance activities on a chosen target are,

1, Time
2, locality
3, visability

With traditional tangible world surveillance proximity to the target significantly effected “visability”, not just by the target of the surveillance measures, but also the “resolution” and “scope” of information available and also the resources needed. The resources in turn were effected by technology available. Similar applies with time and visability.

Which historically ment surveillancec was very limited and held at the Fdet end of the line.

However the development of surveilance technology has increased both spatial resolution and scope to the point where even past time is now decreasingly of relevance, nor as far as communications is concerned locality.

That is “information surveillance” via “collect it all” has created what is a “virtual time machine” where going back in time is simply a case of “pull past records” which are available on near everyone due to “collect it all” and “Third Party Business Records”. The requirment for business records was originaly for “book keeping” and later taxation and legal needs. But this has broadened out in scope as it was realised the records are potential marketable information. And we know they currently being sold via data brokers and given for free to US Government agencies etc.

The problem for those surveiling via “collect it all” “industrial surveillance”, is the “Chaotic Fire Hose” issues. That is not knowing what is worth collecting has resulted in vast repositories of data that is near impossible to use effectively or profitably as it’s a form of “a straw in a hay stack” issue.

It’s regarded by quite a few as “beyond comprehension” but actually it’s not. All you need is some way to find “usefull paterns” or rules and apply those to the data to filter out signals. Effectively this is what LLMs can be very good at, but importantly they can also be very very fast compared to the traditional human analyst.

So LLMs are an ideal “surveillance tool” as a back end to “collect it all” and we know such tools in a more primative form already exisy, and it is the base business model behind the worlds largest private surveillance company Palantir as I’ve mentioned in the past.

The problem with those earlier back end tools was that they were “human dependent” on the generation of “usefull pattens” and many failed due to the “throw the pasta at the ceiling” side issue. But less obvious was that there are limits on how humans can pull signals from noise. In essence we take a linear continuous view and try to augment it by “averaging out noise”. The thing is humans are realy bad at recognising the difference between true noise and the sum of other signals that chaotically look like noise.

This has kept surveillance technology down at the Fdet end of the line even though it has increased in scope beyond most peoples imagination. It is effrctively “stuck on the starting line” where that start line is rapidly growing in length beyond human sight, but is not moving forward.

As I’ve mentiond before LLMs are massive “adaptable filters” that pull human mind usefull information from what looks like totally random noise. But the noise is mostly not actually not truly random or random at all. But it is a tangle of a myriad of signals that are to chaotic for human eyes to pull out.

That is LLMs and similar AI are increasingly getting beter at untangling all the signals and teasing them apart so even the least capable of humans can see all the threads of human society laid out and thus chose which to pull forward, which to tie in knots, cut short, or be totaly removed.

Thus LLMs have in effect have loaded and fired the starting gun, and surveillance is now nolonger bound down at Fdet end of the line, but is advancing quite quickly along that Fdet-Tran line in a hugely broad swath at rapidly increasing pace.

This will be a disaster for Western democratic society and will produce systems way worse than what is currently being imagined in the MSM and trade press for what China and Russia and other authoritarian states are heading / building towards.

But consider, those living in longterm authoritarian states have different societies to the West and as such have already adapted to the instillation of surveillance over several generations. Western democratic free speech society has not. A distinction that tends not to be given the actual consideration it is due in the ICT industry and it’s product users.

Thus by habit those in authoritarian societies don’t leave potentially self harming information around or in communications or their activities in obviously visable ways. Which is not something “free speach” has engendered in Western society.

That is those in longterm authoritarian states have in effect masked their signal with the sum of other signals thus so far have been “hidden from view in the long grass”, whilst those in the West have grown in the sun into a fat crop of wheat…

LLM’s are going to cut through that wheat and long grass alike. Like the ever sharp scythe death is depicted as carrying it will cut for binding all before it. Only Death is an anthropological abstraction of man.

LLM’s will be like those multi millions of dollar “Prairie Combine harvester teams”. Which on Western society that has grown like wheat to the point it is a valuable crop to harvest, it will be rapidly cut threshed and sold off. More surely removed than a whirle wind driven fire storm.

Neither open or closed societies can survive as we currently know them against LLMs effects when a “directing mind” puts them to what most observers would consider “bad use”. Unfortunately human history tells us that this is most probably what will happen even with legislation and regulation to supposadly prevent it.

Because both money and power and what they bring in terms of status and control over others is too desirable for some. Thus they have developed methods to avoid legislation and regulation.