Schneier on Security

Clive Robinson • March 23, 2014 9:51 AM

@ Bruce,

With regards a title you are stuck with the issue of being ambiguous to inspire interest not rejection in as wide a prospective audience as possible thus something like,

Data and the Journey to Power : from micro information to macro control

Or similar.

The most interesting paragraph you have writen above is,

Understanding these trends involves understanding data. Data is generated by all computing processes. Most of it used to be thrown away, but declines in the prices of both storage and processing mean that more and more of it is now saved and used. Who saves the data, and how they use it, is a matter of extreme consequence, and will continue to be for the coming decades.

You have a number of points there which are quite capable of filling books on their own depending on what level you chose to goto.

For instance there are various semantic linkages between information, data and knowledge that can vary wildly depending on who’s view you take [1]. Thus coming up with a consistent view might become an issue.

As Coyne Tibbets has pointed out all activities have information associated with them. The problem is converting this information to knowledge that is then usable to provide leaverage on other activities which in essence is what you mean by power.

As a 20,000ft overview there are four basic stages,

Activity gives information, the collection of which gives data, which when analysed gives knowledge which when used as leverage gives control or power.

From this it can be seen that there is a basic series of work flows,

1, Activity as a cause.
2, Which produces one or more Effects.
3, Observation of effects.
4, Communications of the observations.
5, Storage of observations.
6, Processing of stored observations.
7, Deriving knowledge from the processing.
8, Storage of the derived knowledge.
9, Processing derived knowledge to assess utility.

At this point considerable cost has been expended on collection and analysis but no use has been made of the knowledge to gain leaverage on other activities.

However there is an immediatly obvious problem step 3 “observation of effects” will as I’ve indicated in the past be imperfect. That is of any activity there will be atleast N+1 realities where N is the number of observers each with their unique perspective of the events caused by the actitiy.

But there are less obvious problems with observation of the effects. The most obvious of which is the distance in space and time of the observer from the events caused by the activity. The greater these distances the less use the observation of the effects is, and the greater the subjectivity of the observation [2]. Whilst it is not possible to stop the effects being generated by the activity the distance can be used to good effect to manipulate the subjectivity of the observer thus poluting the information source.

However there is another effect in that as the distance decreases the more effect the observation will have on the activity [3]. Both hard and soft sciences are to some extent aware of these issues but usually they get ignored when not used in a scientificaly acceptable methodology. Ignoring this for whatever reason usually leads at best to usless data and false reasoning, at worst there is little constraint on the harms it can cause.

But to make this worse are the issues of communications (4) and storage (5) of the observations, both suffer from limitations of “bandwidth” that is they can only transport or store a finite amount of information as observed data in a given period of time. This generaly means that information is not only lost as part of the observation and quantatization process into data it is further lost due to the constraints of communication and storage no matter how efficient or inexpensive they become.

Which means that the processing of the data into knowledge (7) is at best an uncertain process, and reliance on it should be treated with considerable caution.

Whilst statistical methods can be used to improve the quality of the data it is very easy to use them incorrectly or not understand the limitations of the results especialy when the data is derived not from information from simple physical near determanistic processes but complex processes with hidden drivers or non determanistic processes.

Whilst the usual scientific method calls for verification of knowledge by experimentation this is generaly not possible in practice with most information sources.

Thus it can be seen that data stores can be easily populated with inacurate or false information.

Which is just one of a myriad of problems for each step in the process that could each fill a whole collection of books in their own right.

Thus I can see filling two hundred and fifty pages is not going to be a problem, the problem is which problems to cover and in what depth, and that is going to be dictated by the slant you want to put on the book.

It’s not clear from what you have sofar said what slant of the many available you want to take, my view would be to pick the slant from the “power” end not the “activity” end and work down not up with a few choice drill down examples. Hopefully you will make your choice early on and thus save time and effort.

[1] I’ve indicated this in the past and I currently hold the view that information is intangible and becomes data when imperfectly quantatised and impressed on tangible elements for storage and communication. As such knowledge is due to the processing of the quantitised data in some manner. One question that arises is “Can knowledge exist without information being stored, communicated or processd as data in a tangible form?” Currently the answer appears to be no from our physical universe perspective, thus knowledge is constrained by the current laws of physics and would thus be finite in it’s totality in our universe. The implication of this is that as knowledge is created within our universe at some point it is also destroyed, which is a problem.

[2] Whilst most people intuativly know the limitations of observations made in space at a large locality difference between actor ans observor, most do not realise that distance in time is actually more problematical. Often you will hear of primary / direct / first-hand observation and secondary / indirect observation. As imperfect humans we reconise the differences in various fields of endevour and use wooly reasoning to set arbitary standards. Thus in law we have “hearsay” which used not to be treated as evidence, but by the addition of “expert opinion” can now be regarded as evidence. However in science the principle of times arrow says it’s dangerous to do this. Because you are arguing –from imperfect observation of some but not all effects that in of themselves age– backwards to a cause and in the process you are in danger of inventing details and constraints which may not be real or relevant. Which is why as a general case “theory” has to be proved by carefull and controled experiment that is reproducable by any interested party. The problem is not only is measurment imprecise it also destroys part or all of the effect being measured [3]

[3] Whilst not immediatly intuative it is now recognised that the measurment of an effect changes the effect to some degree. That is the process of measurment is an activity with it’s own attendant effects that act on the effect being measured. Generaly the smaller the effect being measured and the more tightly coupled the measurment process the more the measurand is changed by the process, and thus at some point the two effects become equal in magnitude and thus a single measurment result is no better than the random toss of a coin. Likewise all systems have noise in some form or another and this also effects measurments as the magnitude of the noise aproaches or exceads the level of the measurand. Thus there are limits to what we can know about a single event beyond which we are at best guessing.