Schneier on Security

Clive Robinson • June 27, 2025 9:15 AM

@ Bruce, ALL,

With regards,

“… but as we start using massive amounts of data to both train and operate AI systems, data integrity will become more critical than ever.”

Err no, “integrity” is the wrong word for what you are thinking.

Integrity is not about the “data” but the transportation of data from point A to point B without change that has meaning to subsequent processing.

The old “Garbage In” has integrity if it does not change hence “Garbage Out” is the integral result.

The problem with AI input is mainly not if it remains integral from the source. But if the input is true / factual that is if the source is hard or soft bullshitting or not.

In science we very much care about “cause to effect” and that it is repeatable and not falsifiable either by deliberate action or faulty / unreliable method. The same is true for mathematics and in both cases the underpinning logic.

In many ways the CIA triad of which integrity is just one component originated before AI was much thought about. With the scope of it applying only to when data had been acquired and converted from the tangible physical world to the intangible information world.

It’s an issue that comes up with authentication by intangible information of a tangible physical object. The integrity of the tangible identification process ceases at the transducer and only after translation gets replaced by the integrity of the intangible information transmission process.

The integrity of the tangible and intangible processes are actually entirely unrelated. And thus the translation process can not carry the integrity across from the tangible domain to the intangible domain.

This has been known publicly in the UK with the “National ID Card” debate last century when Dame Stella Rimington the first female director of MI5 pointed it out to the embarrassment of some politicians, civil servant, and the lobbyists of companies hoping to profit greatly from the introduction of an electronic ID card, when she said,

“My angle on ID cards is that they may be of some use but only if they can be made unforgeable – and all our other documentation is quite easy to forge.”

“If we have ID cards at vast expense and people can go into a back room and forge them they are going to be absolutely useless.”

https://www.thefreelibrary.com/A+REALLY+BAD+IDEA%3b+Ex-boss+of+MI5+slams+identity+cards.-a0138791272

Things that are “quite easy to forge” lack any useful integrity, even though constituent parts might have high integrity, the system always fails to “the weakest link in the chain”.

Clive Robinson • June 27, 2025 8:45 PM

Alan Walsh, Commenter, ALL,

With regards,

“Is this (finally) a useful case for blockchain?”

Probably not, for two important reasons,

The first is, to work a blockchain system has to be mostly if not fully “public”. And as is shown on an almost daily basis the “Confidentiality” or “The C of the CIA Triad” is in most cases more important to users and systems operators than “The I of the CIA Triad” “Integrity”.

Secondly is the cost and efficiency of blockchain systems is very much against their use. And they have very real and very significant issues when it comes to the correction of “human failings” and the resulting legal disputes and current legislation in existence.

Take for instance the European “right to be forgotten” or have records expunged from databases…

But further consider that to be it’s self integral there needs to be three independent blockchain systems arranged in a “voting protocol” format. That will each be far from identical as “transactions in, to records out,” will happen at different times thus orders at each blockchain system. This will significantly effect “The A of the CIA Triad”, “Availability”. In short any such blockchain system will be incapable of “keeping up”.

Clive Robinson • June 28, 2025 2:29 AM

@ Bruce,

In Reply.

You make a number of points and raise a number of questions, effectively starting with,

“If you’re building an AI system, integrity is your biggest security problem. And it’s one we’re going to need to think about, talk about, and figure out how to solve.”

It’s broader than just “AI Systems” it’s true of any system that works on information[1] or just observes it by measurement[2].

Humans have been thinking and talking formally about the integrity of information[3] based on thousands of years of societal grouping, and probably longer than we have been able to store information independently and verifiably of the say of human communication.

In all of that time we have not been able to give information actual integrity, and I think it’s not actually possible to do so.

The fundamental reason for this is information actually has no tangible form that is it is always derived by observation or measurement and as such is intangible or incorporeal[2]. Something that has long been recognised by common law[3].

But importantly all information must be communicable to exist within human comprehension as “knowledge” thus have utility[2]. And as importantly information can only exist in human comprehension by the process of measurement in an environment through observation[1].

As I’ve indicted in my post above,

https://www.schneier.com/blog/archives/2025/06/the-age-of-integrity.html/#comment-446192

The works of Claude Shannon and Gus Simmons show that any kind of “measurement” is fallible as it “communicates information” that has “redundancy” thus “covert channels”. Thus in turn as “Integrity” is very much an observational measurement process it too is fallible and not reliable as a binary result test process.

Thus “integrity” is at best a probabilistic not absolute measure[4].

Further it can be argued as Shannon did for information, that an estimate of integrity is equivalent to entropy in statistical thermodynamics.

Moving on,

You note that,

“We need research into integrous system design.”

Which based on human history and the formation of societies is going to form a new field or domain of research in “technical systems” design (as arguably it has failed in the more general human society).

You outline some areas with,

“We need research into a series of hard problems that encompass both data and computational integrity. How do we test and measure integrity? How do we build verifiable sensors with auditable system outputs? How to we build integrous data processing units? How do we recover from an integrity breach? These are just a few of the questions we will need to answer once we start poking around at integrity.”

The problem as I’ve indicated with security in the past, is that,

“We do not have usable measures.”

In part because,

1, We do not know what we need to measure or why.
2, We do not know how to quantify what we can measure.
3, We do not know how to extract meaning from what we do measure.

As I’ve indicated integrity,

“Does not cross boundaries”

Due to the fact it does not translate into information in ways we can rely upon. Whilst this is an issue for all measurements, mostly the measurements we do have can be reduced down by the laws of nature down to “physical constants”. Which can then be used in a “physical verification process” that can be independently verified and checked.

Thus the question arises,

“What equivalent to a physical constant do we have that we can use as a yardstick to calibrate measures for either security or integrity?”

With regards your question of,

“How [d]o we build integrous data processing units?”

I’ve answered this in the past. The underlying issue is not having anything we can implicitly trust to build upon.

Trust and security are not intrinsic properties of energy or matter we have to build them with components that lack them. This build a property is something we almost take for granted in our technology based world.

In the past I’ve pointed out that we once built fortifications on “solid ground” but as trade became essential we learned how to build fortifications on moving constructs such as wheeled vehicles and then ships.

Ships as such do not have “foundations” and do not require solid ground beneath them. They work by the principle of “displacement of mass”.

Thus the question arises is there a way to get a similar effect for trust and security.

The answer was “yes” and it gave rise to “Castles v Prisons” that I’ve previously outlined (and people can search for on this blog).

Which brings us onto your final question of,

“How do we recover from an integrity breach?”

Well there is good news and bad news on that.

Integrity is about how data is processed rather than the data it’s self.

Whilst there is little that can be done once data has got beyond an individuals control, integrity is something that can be restored or repaired after it has been broken.

Thus Integrity is more like the tap, than the water that can flow through it.

Thus the question arises from that is one of “confidentiality”. Or can we make the loss of Integrity less meaningful in a similar way to having data encrypted can reduce the impact of having security broken?

This is actually difficult to answer because we don’t yet realy have a meaningful measure of what Integrity actually is at a nuts and bolts level.

Thus this is perhaps one of the fundamental issues of Integrity we should be looking to solve first.

As the old,

“You know it when you see it”

Is not really of use.

[1] “Information” has various definitions but one of the more basic is,

“Data that has meaning”

Thus utility, and is derived from observation or measurement by an observer or process. As has been pointed out on occasions data[2] has levels of meaning or context, thus we have,

1.1, Raw data from observation
1.2, Meta-data which is effectively “data about raw data” and gives observation meaning.
1.3, Meta-meta-data which is “data about processes of obtaining Meta-data”.

Thus arguably data is “turtles all the way down”.

[2] But what is “data” well in a circular argument it’s information that has utility. But more usefully it’s the result of making measurements, that can be,

2.1, Communicated
2.2, Stored
2.3, Processed

However it can be seen that each is dependent on what proceeds it. Thus to process information/data you must be able to both store and communicate it, and to store information/data you must be able to communicate it.

As such we generally consider communication of information/data to be by the movement of matter or energy under the influence of a force. Where the energy or matter has information modulated or impressed upon it.

However the process of modulation or impressing allows us to see information/data as being separate from energy and matter and the forces and limits that act upon them, so treated not just abstractly but as a separate entity. In effect information has an incorporeal form[3], thus can be treated mathematically in it’s own right. Hence since the late 1950’s we have “Information theory” started by the work of Claude Shannon and others during and prior to WWII.

[3] Information / data as an “incorporeal form” has meaning beyond mathematics and the laws of nature. Since “The times of the Tudors” it has had “legal status” via “letters of Patent” and earlier “common law”. A result of which is a written document can have a status equivalent of a “natural person” hence in treaties, legislation and regulation the expression

“Any person legal or natural”

Or equivalent can be found.

[4] For those not familiar with absolute and other measures it’s easier to provide a link,

https://thisvsthat.io/absolute-vs-relative