Schneier on Security

Clive Robinson • January 24, 2024 6:03 PM

@ Bruce, ALL,

Re : What progresses so can trigger.

In the example they use a date as the trigger or tipping point.

The aim being that the bad behaviour like that of certain automobiles does not show up during testing but when on the road.

Time is effectively monotonic and progresses in the forward direction (ie only increasing) and all it’s derivatives likewise. One concequence of this is once past the trigger point you can not go back.

As I sometimes note,

“You are only a murderer when you have both killed and been convicted”

But I seldom mention the rider that,

“Once you have killed you can not undo it, but convictions can be appealed and convictions reversed or pardoned.”

Thus whilst you can not go back before the trigger point the concequences can.

This gives us three basic models to consider for “bad” behaviour.

1, Bad responds to a non monotonic trigger (automobile testing).
2, Bad responds to a monotonic tipping point (as in the paper)
3, Bad only responds to a non monotonic trigger after a monotonic tipping point.

If the LLM weights can not be analysed directly for behaviour as we are told is the case currently (in part because the network acts like each node output acts like a oneway function).

Then bad behaviou may only show during initial testing in the first case.

In the second case bad behaviour won’t be found during initial testing but may be found during testing past the tipping point.

In the third case bad behaviour won’t be found during initial testing and is unlikely to be found in testing after the tipping point.

Thus finding bad behaviour needs upto three things,

Firstly a test is needed that will demonstrate the bad behaviour.

That secondly can not be detected as a test rather than normal usage.

That thirdly is used after any tipping point.

The thing is whilst time is easily seen as usefull to define a tipping point there are other monotonic functions that could be used.

One such is the number of uniquely determinable users.

Another is the total of user queries

And others such as the type of query sophistication advancment by users.

Another is “not in the model” queries. Humans progress and culture moves forward in terms of language. One such is when a word changes it’s usage. Such as “Thatcherism” from a persons name to there apparent behaviours that are either new or sufficiently notable by society. Whilst they can not be in the model initially they will show up in logged user queries.

Thus the question arises as to can any monotonic function actuall be predicted thus have a test for?

Further will such monotonic functions evolve as a result of models being updated and any bad behaviour be a consequence of that as well?

As I suspect this is the case it requires testing to be continuous which is a significant cost on resources.

In the past I’ve talked about “Probabilistic Security” where you make a concious choice of how often you carry out testing as a percentage of system resources usage.

But I also talked about process signitures as a sign of malware or other unwanted / bad behaviour.

This has an upside and a down side.

If the system can not detect you testing it then there can be no anti-test patterns (which is why I designed C-v-P to have a hypervisor that actually halted the CPU and took over bus control to examine the memory etc and the CPU to only have access to it’s internal process time “ticks”).

With AI of all forms we now have to consider it a “hostile insider” and mitigate where possible accordingly.

This is going to necessitate a radical rethink in the way we do security…

One such is to use “voting systems” that is you would use say three LLM’s that have different but complementry input data subsets randomly built from a large data set. Look at it like “double blind testing”. It’s not going to be perfect but it’s about the best we can hope to achive with the current point in human knowledge and understanding.

Clive Robinson • January 25, 2024 9:09 AM

@ R.Cake, Billbo, ALL,

Re : Systemic risk from endogenous risk.

As they say “this is not my first rodeo” and I’ve written a few pages on the subject, so this is a synopsis to what underlies your point of,

“The product may still be fully compliant to the original requirements specification, but alas this specification was written only by humans…”

Having had involvment with many different types of specification in my career I can confirm that,

“Imprecision is a major but unavoidable issue with specifications.”

And the “imprecision” and “unavoidable” applies to more than just language. Thus the question of “Why?” and what is fundemental to it.

It starts way before “specifications” arguably it starts long before any project, process or “System” of any type including the founding of the organisation(s) that manufacture “the good or service end product”…

As a rough rule of thumb if asked people generally assume that projects have hard start and end points in a fixed time frame and come from nothing and develop to full deliverable by a “fully found” process.

That is, the process is “Discreet, Bounded, Correct, Fixed, and Ordered” such that it is “fully determanistic”. Nothing could be further from the truth…

We have found from actual observation and enquire that all projects, processes and systems even those with no human or other organic life involvment have three fundemental asspects,

1, Entropy.
2, Organic Growth.
3, Evolution.

And although we have no idea why and probably never will none start from nothing, and the three fundemental asspects appear to be the “built in before it started” rules or laws of the universe. And beneth those the true “Three R’s” of the universe

4, Random (stochastic)
5, Recursion (infinite)
6, Recognition (selection function)

That underpin reasoning. And beneth that the two fundementals of physical (energy/matter) every objects has,

7, Motion
8, Direction

From the application of four basic

9, Forces

As seen through the work of Newton[1] and later Einstein.

Every where we look, every process or system we’ve studied or has occured that we can find record of.

Even the “Fundemental Law(s)” of the universe,

“In statistical mechanics, the laws of an underlying physical theory are used to determine the dynamically possible trajectories through the state space of the system”[2]

In philosophy back just after WWII and with the nuclear bomb having impirically proved many things especially about projects, it was famously suggested by Goodman[3] that,

“There is a connection between lawhood and confirmability by an inductive inference.”

Which some philosophers for the sake of argument say,

“May be a case of the observer effecting the outcome of the process.”

An unwarranted over generalisation of the “observer effect” of physical measurment that was fasionable in soft science and other knowledge domains a decade or so ago[4]. But it implies,

“All pots are watched some of the time.”

Which would further imply,

“An immortal omnipresent observer with the ability to blink/look away”…

Thus reduces to circular reasoning based on a fundementaly stochastic process.

Similar reasoning brings you back via the three R’s to the three fundemental aspects of entropy, growth, and evolution.

Entropy has two basic meanings that are the flip side of each other,

To move from “order to disorder” of determanistic to stochastic, and thus to increase possabilities or information capacity.

You can see this with Lego bricks. If constructed as a determanistic object like a sphere there is not a lot you can do with it. However break it up into a random pile of bricks, you have seemingly endless posabilities to create other objects.

But you don’t move instantly from one object to another you first have entropy to break one coherent object down to give you the building blocks to grow another object.

It’s the point where most people consider a project has not yet started.

To go from a random pile of resources to a new object requires that it grows by the bricks being put together.

However the growth process is limited by both resources availability and type, and the constraints of the likes of “the laws of nature”. But there is still a lot of freedom there.

But for any desired object you don’t just randomly join those bricks together you need an idea of what that object should look like and that acts as a further constraint. That is it is a “fitness function” random can happen but only within the limits of the constraint.

But the problem with this is that it can cause further constraint. That is if you start with using just one type of the many brick types you have the supply will be exhausted before you finish building the new object. Thus you develop a new set of fitness functions during the growth that gives you your evolutionary process.

Get the right fitness functions applied in the correct sequence and you end up with the desired product.

But on that evolutionary growth you can make a myriad of mistakes.

Many projects in the software side of the ICT side of the industry are “not to plan” because as the old military warning says,

“No plan survives first contact with the enemy”

And the enemy for software development which is effectively always “first of a kind” is “the unknown”.

So you “make it up as you go along”

All projects that are “first of a kind” are “Development Projects” thus not determanistic, but random choice constrained by fitness functions. Thus both the product and the fitness functions “evolve” during the growth of the project. Once a product is developed however it can then move into a more determanistic production phase where most of the fitness functions have been found and codefied into a plan. But resource or “feed stock” issues can happen thus new random choices constrained by old and new fitness functions happen.

Unfortunately the creation of new fitness functions is often predicated “down the line” by an “end stop issue”. That is a random choice may not show up as wrong at the time of the choice but a very great deal later when something “hits the buffers” or “hits the end stop”. This nescesitates winding things back and changing the choice and “reworking” the production process. From this a new fitness function is derived.

What seldom happens in software development is “History Files”. Where what went wrong, how it was recognised, how it was fixed and how the resulting fitness function was developed is correctly, formerly, recorded, and published without prejudice so it can be learned from.

In science and physical product development and production History Files are mandatory under the rule of,

“If it’s not written down it never happened, so you don’t get benifit from your work.”

A side effect of “No history files” is “no evolution”… One side effect of which is “a veritable tsunami of technical debt” where sometimes it’s only solved existentialy by insolvency, bankruptcy, and demise of the organisation, with attendant layoffs etc.

The excuse for such behaviour boils down to an idiot neo-con incompetent manager mantra of,

“Don’t work harder, work smarter”

Combined with another idiot neo-con mantra of,

“Don’t leave cash on the floor”.

Hence errors do not get recorded, lessons don’t get learnt, mistakes are repeated ad infinitum, costs rise uncontrolably, and those with foresight and the ability to excercise it, “jump ship” rather than “go down with it”.

The lesson is,

In any development project growth is organic and errors are a consequence of open choice where there is no past guiding information/knowledge. Unless constrained by fitness functions of evolution, errors will rise proportional to a power of the number of objects and their linkages.

This applies to all areas of a project, process, or system, thus existential errors are a fact of life. It’s where “endogenous risk” comes into play, that is how people manage errors, by recognising them, correcting or mitigating them and most importantly learning from them.

As I note from time to time in ICT we don’t appear to want to learn from our history.

I ask you is that realy “working smarter?” as many neo-con thinking managers see it…

[1] What Newton gave us was a universal reductionary process to describe what appeared to be the complexities of “The Natural” by reason rather than using approximation by “curve fitting”.

Newton realised that objects ‘O’ have effects ‘e’ that from earlier geometry, the magnitude of which would diminish with the radius distance ‘r’ at the inverse of the distance squared,

e = O/(r^2)

And further that the objects had properties ‘P’ that had atributes ‘A’ in common thus

e = A.P / (r^2)

So he reasoned that the magnitude of the effects that result between two physical objects would have a combined effect that was multiplitive so

E =(e.e) =(AP1.AP2)/r^2 = A(P1.P2)/r^2

Substitute in masses ‘m’ and the gravitational constant ‘gc’ and you get the combined gravitational force,

Gf = gc (m1.m2/r^2)

That we should have been taught in high school along with the fact that the form of the equation derivation is general to all forces.

So we can using further basic geometry of Pythagoras end up with a formular for elipses, that rearanged happens to describe orbits that are the fundemental description of two moving objects in close physical proximity without other forces applied. Using the same derivation method you can apply another force such as friction or wind resistance and so describe the path of a fired arrow or cannon ball or just a thrown object.

Thus you end up with a reductionary process where apparently complex behaviours can be simplified and understood within the limits of what we can measure.

[2] Roberts J., 2008, “The Law-Governed Universe”, Oxford University Press.

[3] Goodman N., 1947, “The Problem of Counterfactual Conditionals”, Journal of Philosophy (44:113–128).

[4] It’s unwarranted because of the confusion or conflation most have over measurment and observation. And an example of linguistic issues you refer to. Because it’s the actual process of “measurment” that effects the process, not the looking at the meter of “observation”. More fundementally and simplistically in the dark you shine a light, the photons –fundemental EM charges– of which hit physical objects at the speed of light thus by a force imparts energy to the object. It’s that force/energy acting on the object that effects the experiment, not the fact you may or may not see the reflection or reemission of photons that have come back from the object towards you at the speed of light. To say it’s the “observation” not the “measurement” implies “time travel backwards”.