Schneier on Security

Clive Robinson • March 5, 2010 10:09 AM

@ Winter,

I think that you misunderstood what I was getting at.

The value of 13million bots is way way higher than the current market price. That is even a one off transaction could if carried out the right way net 0.5BillionUSD at quite low risk to not only them but the botnet as well.

I note from your reply to Seth,

“Either their market is simply not large or liquid enough to supply such a ROC. I think their market failed largely because of transaction costs (finding and convincing “honest” clients).”

That you are thinking similar thoughts.

If it was me I would be looking at having a “covert cloud” ontop of the botnet doing highly parellel tasks that have a high dollar value either directly or indirectly.

For instance what is the value of the private key to sign code for protected devices (mobile phones / DRM etc).

There are a whole host of other things like this that don’t realy put the bot net at risk or for that matter the operators.

Using such a resource for spam or DoS shows a real lack of ability in the thinking department for these people.

Oh and they don’t have to find “honest clients” but likewise with a little thought the bot net could be laundered for certain types of “cloud” activity if they went about it the right way.

Clive Robinson • March 5, 2010 12:06 PM

@ Brandioch Conner

“And yet, just a few weeks ago, experts on this very forum claimed that taking out a zombie network in that fashion was impossible.

And yet it was done.”

Err no it has not in any way been taken out. All that happened was the control channel was disrupted so the bots where not getting instructions.

According to one artical shortly after the control channel was disrupted one of the bot operators bribbed a person working for an IP registry and got part of the control channel back and then turned those bots onto one of the organisations as a DoS taking out not just the company but the ISP and all the other companies that used the ISP.

Worse is Spanish law does not actually prohibit what has been done, so the gentalmen concerned are out and about and free to download the rest of the data they aquired and stashed away.

Realisticaly it looks like they might not even go to trial let alone see the inside of a jail. The Police are looking for evidence of a crime (such as Fraud) to actualy charge them with and from what has been said it looks like that might not even happen.

Which leaves the point so delicatly put by @ sebD,

“Sorry if this is a dumb question, but what happens with a botnet once the people behind it are arrested ? The computers won’t be cleaned of the malicious code. Is it imaniginable that someone takes control over it ?”

In this case the original bot net owners got some of it back.

However the rest of the net is still (in most cases) sitting their waiting for instructions via the control channel.

There is of course the question as to if the operators put a “deadman’s switch” in.

As it was a “fire and forget” that had a vector to get across air gaps I doubt that it has a switch and if it has I again doubt it has a “take the world down with it” payload.

So the chances are some of the bots will sit there looking for the control channel untill the “owned PC” joins the “great bit bucket in the sky”.

Clive Robinson • March 6, 2010 1:38 AM

@ Nick P,

Carefull what you wish for 😉

I have in the past noted that you would be safe using a non mutable (ie CD based) OS and no sem-mutable memory. Thus power down and any problems go away.

Well for various reason we have discussed I was “over optomistic” on the degree of “safe” and should have said “posibbly safe If, you also…”

The one thing this bot net did was use a very old attack vector of virus style infection from removable media….

Thus it could cross the “air gap” from your Internet PC to your private PC and get at your confidential information…

Thus it only required two other things,

1, Getting control information across the air gap.

2, Getting the confidential information back across the air gap.

Humans are creatures of habit so if you cross the air gap once the chances are you can do it twice etc, so getting control data in is almost easier than getting the malware in.

Oh and if you think about it as long as the removable media is infected the “Internet” PC will just get reinfected the minute the media gets pluged in again…

Thus the hard part is hiding the confidential data going back to the Internet either on the removable media or the low(ish) bandwidth comms link.

Saddly not all (read most) users can be educated to this fact at a level where it modifies their behaviour.

But there is the issue of AV and zero day…

Malware writers are getting smart faster than almost everybody else.

If they first stop and think how to capitalise their malware properly (ie not Spam or DoS) and make the bot net properly covert then we are potential going to “be living in” some seriously “interestingly times”…

I just wish a few people would wake up to the fact and start thinking seriously about it instead of blowing their trumpets and resting on their laurals.

Clive Robinson • March 8, 2010 12:12 AM

@ Nick P,

“Additionally, we could combine my Castle and embedded server board ideas with your prison view to use a bunch of small, embedded Aegis-style chips running in parallel. The totality of these, along with a Master node, become the system.”

The hardware sounds like it’s as close as it’s going to get without going fully custom (not that that is to much of an issue these days).

From the OS etc side if you say your “castle” is the “Prison Gov & Warders” then you are 99% of the way there, but coming from the top down not the bottom up.

The bottom up aproach adds some interesting twists that improve on what the top down aproach will give you.

And oddly with the bottom up aproach it’s all mostly for free because of engineering in the “sweet spots”.

First a little boring history as to how I arrived at the basic design.

The idea all started with a thought almost identical to your,

“I don’t see regular code monkeys doing it right or even caring.”

I’d like to say it all started at a security event for programers a few years back however the idea had been bouncing around my head one way or another for quite some time before that (actually nearly 20 years ago when I was thinking about doing a PhD. But could not find “Readers” to act as supervisors because they didn’t get the problem…)

Any way back to the security sminar which was little more than an informal side event at another larger event.

The bod out front asked how many years production application programing time people had had, and it varied but averaged around 12 years across something like 50 people. So something like 600 years nose to the grind stone experiance give or take a little.

The bod out front then asked how many had had formal security training. Lets just say the total was around 2 days amongst the 50 or so production programers there…

So ~2days in 600 years in production code cutting or a ~1 day in 100,000 days (most progs work more than 7 days a week equivalent even if not paid for it)… And the programers at this walkin session where there because they had a personal incentive to be there. I dread to think what the rest of the industry average was like.

Yes it was starting to change more recently but then, well first some major companies and then the banks gave the world the one finger salut and in the resulting economic climate it has regressed more than ten years in many respects in about the length of time it takes to blink twice (or atlrast thats what it feels like).

Thus you can probably sympathise with my view that it is a waste of time trying to resolve the security problem any time soon by education of “regular code (cutting) monkeys”. Irespective of if they want to make the effort managment etc are just not going to let it happen, 99 times in a 100 the urrent managment view is “liability mitigation” not “security”, (which means the money goes to auditors).

I’ll admit I’m perhaps a little jaudiced but on the “computing side” I come from a CLI and Embedded background. Unlike most code cutters I’d had to cut my own BIOS / OS etc as well as the application on top and get it out the door with “no patches possible” (yup that’s the way it used to be ;). We had to be engineers not artisans or artists.

Also due to when I entered the industry I’d designed the actuall logic circuit guts of what we blythely call CPU’s, and cut the state machines and microcode to bring them to life from a handfull of 74Sxx chips (anyone apart from me or NASA remember the 181?) or ECL.

Basicaly I’ve lived with abstracting design up the stack and seen how the faults slide in proportional to the abstraction and now get left to “patch tuseday” etc (and lets say I’m less than impressed with it).

You may know the current 10/20/30/40 rule of “application code error causes,

10% = Business logic errors.
20% = User input errors.
30% = Data timing errors from back end systems etc.
40% = Unknown “glitches”.

The industry clearly has significant growing pains from the Terminal/CLI days. Only 30% of the errors are down to “human issues”, 70% are down to code cutting issues. Much of which is due to code cutters not understanding “state” and “timing” or they and others how to “Stress-n-Test” for them (regression testing only catches “Known Knowns” and a few “Known Unknowns” as for probablistic testing or “fuzzing” that’s what the users to do better…).

If you reflect on this sad state of the idustry and ignore the needless bells and whistles of the GUI etc for a minute you will see in many ways we are a lot lot worse off than the old “Unix Scripting Methodology” days. Mainly because we have very much got component re-use wrong, not just from the security asspect, but in general.

However also reflect abit on the Unix method and you will realise it is about as far as we’ve got in realistic “security seperated” code re-use.

You have three parts, the OS provides the plumbing and the CL utilities provide the base functions, and the script provides the overall program logic.

You see a simple but elegant split of duties with it. You have a script telling the OS to line up functions and the required plumbing to push the data through to achive good solid results.

So good infact that some systems that used the method to “Rapid Application Design” found they did not actually need to code much up (and in the case of many SysAdmins they had better things to do with their time).

In fact a new generation of Unix scripters came along with early web Common Gateway Interface (CGI) scripts for early web design. And guess what we are going to see similar pop up again with Cloud Computing.

Scripting this way is interesting because of the seperation, and how it also alows parellel threads to run etc.

Impoertantly though the OS takes care of a big chunk of “State” and “timing” which the script writer usually does not see or care about as it just happens. The (apparent) limitations of the CLI functions force the script writer to remove (needless) complexity and (usually) deal with exceptions. Oh and it is usually fairly clear at what point in the script things break thus making testing easier. All of which are good 😉

So what about “security”,
if you think about it as long as the OS does the plumbing and data access securely then the script writer does not need to know about the low level security only the high level security. Thus a big chunk of the problem is taken off of them and they can get on with the business logic etc (saddly Unix security sucks by current needs and it needs addressing).

Thus you have small generic functions (ls / tr / grep etc) that act like filters. They have no need of security in general because of the way they work. That is the OS hands them data they filter it and hand the result back to the OS.

It is the OS that actually does the security via file permissions etc.

However there are issues with this the first is the Unix Security model is way way to granular and way way to limited in scope (though some tools etc have improved this quite a bit). The second is the functions have become bloated and overly complex for what they do 99% of the time and that is bad for security.

It also gives rise to huge globs of needless code which makes things oh so inefficient.

So sadly the code cutters won the day by weight of numbers and other scripting methods with less seperation etc where brought in. And some have been very very successfull but done baddly.

We see this problem with the proliferation of code libs that are available (have a look at the Perl CPAN resource or the Python equivalent to see what can happen with scripting libs).

Unfortunatly we have lost the seperation and thus the security and other desirable asspects in most cases 🙁

Now I have also had cause to think a little on Turing and Goedels little problems (halting and undecidability etc). And realised yes you could do an end run around them…

The problems arise mainly because it is a “single” universal engine “observing it’s self” effectivly “from within”, and there is no formal seperation of code and data.

So I thought OK lets keep the universal engine with all it’s faults and many many advantages BUT we would add a second non universal engine to keep an eye on it.

How does that change the equation? Well actually in a limited domain such as a security hypervisor it changes it a great deal (it does not 100% solve it but that is another issue).

For security this second engine does not have to be universal at all (a simple “all states known” state machine will do). And you can make it in such a way that it always fails safe. You can remove “general looping and branching” and in many ways make it very like a DSP system that processess data but does not act on the data except when a limit is reached in which case it fails to safe (ie halt in this case). Oh and the data it processes is not the script data but the signitures that arise from the functions of the script executing in the Universal engine which is why a DSP like system is fine.

Further you do not have to use the rather useful von Neuman architecture, a simplified Harvard architecture is actually better for the job. Thus you can further seperate code and data compleatly (under certain constraints).

Thus you have your von Neuman universal Turing engine being watched by a subset of the Harvard architecture which is also a limited not universal engine.

Thus Turing and Goedel still get to play in the universal engine but Einstien belives his determanistic God of the Harvard engine is watching over them (he still gets to play dice though 😉

Thus you move from the notions of “security is not possible due to Turing and Goedel’s problems” and “security is only possible in a fully known state machine” to the working middle ground.

Mathmeticians have already done this with a similar problem at Los Almos where Monte Carlo methods came to life (so the idea has a pedigree 😉
Thus you could call it Monte Carlo or probabalistic security.

So much for the over view.

1) Now if you simplify the view point a bit there are two basic types of security breach issues based around the inherant faults of the von Neuman – Turing engine,

1.1, It occures by accidental combination of faults.

1.2, It occures by the deliberate excercising of faults.

The second is AKA Malware.

Take a leap of faith for now that the accidental faults can effectivly be stopped by the system I’m outlining, I’ll come back to it later.

The issues with Malware are actually quite simple to describe it is “extra code” that “should not be there”.

2) Malware comes in two basic forms,

2.1, Built in.
2.2, Later additions.

The first I shall call “Malicious code” not Malware. The distinction is important because of the assumption that the code is “known to be good” when in fact it is not. And it is the main reason the current fad of “signed code” is currently a bit pointless (except where pointing the finger is concerned).

Malicious code “should be” detectable / preventable to a certain extent by formal methods used in the design and build. But it’s early days on this and there are a lot of steps between current formal methods and code signing which are not covered thus allow plenty of scope for malicious code to be slipped in unnoticed.

Malware is what attackers add to a previously “known good” system to excercise inherant faults.

Importantly it is tangable “extra” code.

3) For this extra Malware code to survive even casual observation it has to be able to hide,

3.1) Malware is usually “trivialy” easy to find on a system that is not running. Simply by checking mutable memory (file systems etc) signitures etc. That is, it is “trivial” to determin on a byte by byte basis not to say it is “trivial” in time or work load. The exception of cause is those dynamic “pesky user” files…

3.2) To hide on a running machine Malware needs redundancy in the system it is on to set up what is in effect a Virtual Machine.

4) Two points arise from this Malware VM view,

4.1, If it cannot have the required resources it cannot hide.

4.2, No VM is perfect, time based and other signitures will reveal it’s existance.

5 ) The less resources Malware has the easier detecting it becomes.

That is because it has less ability to camoflage it’s self, and detect probing and fake responses.

Importantly this is not a linear trade off which means there are sweet spots to be exploited.

And this is why bottom up as oposed to top down has advantages because in general it’s at the bottom where you find the best sweet spots.

[To see this think of car safety, that is a car seat belt is only “so so” you can spend more money but each linear step in safety comes at a power law or exponentialy increasing cost. Sometimes the cost curve is so steep it might as well be a brick wall. Likewise with an air bag.

However use both an air bag and a seat belt you get improved safety.

But… You can depending on the curves get improved safety using less expensive seat belts and air bags. And if you design them into the system from the start you effectivly get your improved safety for free or less…

You can do the same with other safety features like crumple zones and SIPPs etc. Thus get a quite dramatic increase in safety but for no additional cost.]

You just have to work within the design sweet spots from the very early stages of the design to get the best advantages by makeing the appropriate trade offs.

6) One trade off is if you take your program appart into little tasklets or fuctions etc the resource required for each part is fairly easy to calculate in most cases and there are libraries of code already in existance for parellel processing to show how to do this effectivly.

Thus, say you have 20 little von Neuman – Turing engines, you can limit the amount of RAM or CPU cycles or input or output, for each one to give it just sufficient resources to do the job and no more.

Thus each is a jail cell tailored to the requirment the function running inside it and very little more.

This put’s preasure on the Malware in more than two ways.

6.1) The first is it does not have sufficient resources in any cell to be able to achive very much let alone hide via a VM.

6.2) The second is it cannot unlike the desired program communicate from cell to cell simply.

6.3) Thirdly if the tasklets are sufficiently small their complexity is low thus their charecteristics are much more easily calculated thus they can be for the whole program.

6.4) Forthly in a scripting language the functions can have base signitures without knowing what programe is going to use them. As long as certain bounds are known just before they are used the signiture can be calculated.

7) Due to this the programer does not need to know what the signiture is or care, they just have to know what the bounds are and pass them to the fucnction control channel.

Thus the OS tasking the script has three parts to it that. The first traditional Unix two that,

7.1, load the function (puts the tasklets in the jails).

7.2 Connects the plumbing between the functions that the data traverses.

Note I don’t say objects for good reason, as most object implementations are designed to be efficient not secure, and thus can leak/mix code with data etc. They also tend to be overly complex as well…

It is the third bit where the security happens.

7.3, Each function sitting in it’s jail is fed data by the OS has and has results taken away by the OS as normal. However unlike normal the OS also monitors these channels to ensure they stay within the security signiture bounds.

Further every so often the function is suspended and the jail is searched for contraband (ie extra code or data that should not be there).

7.4) The price that has to be paid is basically in “cheap” CPU workload not in “expensive” programer effort. And CPU ability doubles about every year for the same cost whilst programers want pay rises each year…

8) Now this brings us back to the original point,

What is the difference between the Castle and prison.

Actually very very little they both have,

8.1, A person in charge (Duke/Govnor).

8.2, Trusted staff (Knights/Wardens).

8.3, Restricted enviroments where limited trust workers work (Serfs/Trusties)

8.4 Closed environments where untrusted workers are kept and worked (slaves/prisoners).

Further they both have external and internal defensive systems.

9) The real difference between the two is in the tipping point for trust, and the basic design that follows on from this.

9.1) In a Castle most and sometimes all the people inside are trusted implicitly and interact freely in well resourced areas (which is the analagy of our modern office networks).

9.2) In a Prison most are untrusted are not allowed to interact at all and have very few resources.

9.3) In a Castle the security is by and large a perimiter issue.

9.4) In a prison security is mainly an internal issue.

Thus in a castle a low % of resourses are devoted to internal security and a higher % to external security and creature comforts.

In a prison the opposit is generaly true.

10) The trust tipping point also changes the view point of the internal resorces.

10.) In a Castle you look at cohorts of considerable numbers of men who can be trusted to use the resources at hand wisely to get small and large jobs done and they achive more per man for it, that is they are more efficient.

10.2) In a prison however you look at individual or one or two prisoners in a cell, their available resources are just sufficient to carry out small simple tasks and no more, and they are heavily monitored.

Thus it is the trust tipping point and granularity that make the difference between the Castle and the Prison.

In your Castle view the tipping point is near the OS level, thus the processess are like “cohorts” they are efficient but they are difficult to check for security.

In my Prison view the tipping point is down at single functions, thus they are like “prisoners” they are compleatly untrusted, monitored and only given sufficient resources to do their job.

And although less efficient the prison view is much much more secure.

11) At first sight less efficient would indicate higher cost.

However is that actually an issue?

As I noted above in 7.4 you are actually looking at CPU workload -v- software development cost.

11.1) Most if not all systems these days have CPU cycles to burn, thus they are already grossly inefficient so the cost there is effectivly less than zero.

11.2) The system should have minimal impact on code cutters, and may well enforce “de-complexity”

11.3) It will improve exception handeling and fault reporting thus easing not just test but field maintanence.

11.4) It will make users feel “enabled” and thus more responsive to the system.

11.5) It will also enable easy migration onto cluster style systems (think Cloud etc).

And quite a few other advantages not least of which is the increase in low level security against malware.

11.6) The downside is there is currently little hardware to support it. However as you noted above the chips to do it are starting to appear for other reasons. And the systems they end up in may well do “very nicely” as blades or COTS boards.

11.7) Ovarall the idea has more benifits than costs even without the increased security asspects. When the increased security is taken into account it has significant advantages for similar cost of existing systems.

12) Now a prediction or two for you that also tells you why I’m looking at it currently.

12.1) History shows us we go through computing cycles from central resources with user terminals/thin clients to self contained all in one desktop/laptop systems and back again.

12.2) We are currently swinging back to thin client with cloud computing etc.

12.3) CPU manufactures cannot alow this to happen as the development costs for low demand server CPU’s is significantly offset by high demand client CPU’s. If the world becomes too server centric then progress on CPU’s will stall.

12.4) Thus what is likley to happen is that our migration to the cloud will continue and $aaS (replace $ with appropriate letter) will first be outsourced then brought back into house as the silicon venders move the tipping point to maintain revenue.

12.5) Along with this tipping point will be the putting of server type CPU’s into more desktop and laptop systems.

12.6) But the spare CPU cycles will not go to waste as they currently do. The cloud will migrate back onto these platforms in a couple of ways.

12.6.1) The first will be the equivalent of SETI@home.

12.6.2) The second will be web servers on the client machine either as middle ware or middle ware and backend.

Such is the nature of re-use.

12.7) Cloud “Software as a Service” (SaaS) will see a strong “thin client” development for smaller more portable “connected” devices (netbooks mobile phones etc).

We have seen the start of this with Google’s two OS offerings (Android and Chrome).

By current necesity the security of the client will get a lot lot stronger (something I started banging on about ages ago and Google actually got on with on the quiet). OS level security will migrate up into the browser, or conversly the browser will migrate down into the client OS (either way should improve security if not done the MS way with IE desktop intergration).

My money (and Googles) would be on upwards security migration with the web browser becoming the equivalent of the new user “graphical shell” and comms and data flows migrated back down into the OS to get “security seperation”.

12.8) Thus we will actualy favour hardware based VM’s and MILS kernels and also encorage them to become finer grained.

12.9) The question then becomes will the kernels split out a large amount of functionality into “user space” with the equivalent of the old Unix CL utilities. Simply by splitting out the “trust” into the kernel where low level security actually belongs (not in the apps).

12.9) If this spliting out does happen will the dual path (user data / security control) in the kernal and functions be implemented giving the “signiture” based probablistic security.

I hope that adds some background and clarity to the idea?

Clive Robinson • March 10, 2010 7:09 AM

@ Winter,

“A very nice write up about castles, prisons, and compartmentalization. Most certainly a reinvented wheel, though.”

Actually not, read on to see why.

“Did you ever look at Bitfrost and the XO? The Android security model? They are going that way, and have implemented it.”

As far as I remember the One Laptop Per Child XO model is based on BSD “jails”.

I’ve had a look at the Android document you linked to and it appear Android has a similar problem.

They are both “top down” approaches to security and both stop their granularity at the process.

This lack of granularity makes generating signatures and controlling resources difficult as those who have played with SELinux are more than fully aware.

I have actually proposed two parts to the hardware “cell” which is a point you might have missed,

1, Minimal resources in hardware.
2, Signature control by hypervisor.

Likewise the program is not dealt with at the coarse “process level” nor the less coarse “thread level” but the fine “function level” (it could be done at the various levels under that such as the high language statement level or even the byte code instruction level).

The point is that a “cell” limits resources defined on what the function is to do and how it goes about doing it (the functions are built in like the old style Unix CL utilities and thus have well defined actions and fairly easy to produce signatures).

The resources made available to the cell via the hypervisor are,

1, CPU
2, Memory
3, Buffered File I/O

The function has no knowledge of which process it is part of only what it has been invoked to do via the script. The Script passes bounds to the hypervisor that tell it what the function should be accessing.

The hypervisor knows from this information what data to send via buffered file IO and what data it will get back. It will also know from the static analysis of the function what the CPU and memory parameters are.

Thus any malware has to fit inside of this very resource controlled cell and communicate via the controlled IO whilst still staying inside of the signature mask.

Further at any point the hypervisor can stop the function without it’s knowledge and do a “cell inspection” to check what is there.

Thus the malware is going to have a very very difficult time (not impossible but getting close to it).

The advantage of this “bottom up” approach is it gives you a whole load of extras as well for free.

You can see that various researchers are nibbling away at the sides of the idea but they are still coming at it top down not bottom up and thus are constrained by their own outlook.

Oddly if you look at the Android paper you linked to, sections 7.15 and 7.16 are very close to the idea but just don’t quite go the distance. The reason being I suspect is it would require a whole re-work to Android that is not going to happen.

The important thing to remember about the hardware hypervisor is it is not like a traditional von Neumann architecture but a more limited subset of the Harvard architecture somewhat similar to that you find in some DSP chips without going into the ins and outs of it this chip sits their as a warden looking at the cells. If it detects anomalous behavior it locks down the cell and passes it up the security stack.

One advantage of this Script/function approach is that it allows the functionality of unused parts of the program to be effectively removed. Thus for argument sake you have a music player that goes to the Internet to download track and title info etc. You can simply say to the security system not to give any resources to that particular function and for it to return a null string etc. Provided the code cutter who wrote the script followed proper exception handling then the script should carry on without it or if the function is vital for some reason, stop the script and produce a meaningful error message.

SELinux can do coarse grained resource control but it is only at the process level which means it cannot do the above unless it is built into the program which we known your average code cutter will not do simply because of “efficiency and features”.

I’ll leave it up to you to decide if my “wheel” is engineered for a function or artisan made for any horse drawn cart that comes along…

Clive Robinson • March 12, 2010 9:34 AM

@ Nick P,

“I swear you might as well be talking about MILS. Your prison approach & their castle approach parallel so much.”

Yes and no MILS has similar granularity.

The bit it lacks over and above the “DSP” bit is having the pre built “function” components with pre built signitures to monitor.

One of the points behind this is to remove a big chunk of the low level security issue from the equation (ie code cutters).

The programer (much as in the old IBM assembler) selects a function to do something such as read in data from a file (or the Unix equivalent etc).

This function comes in two parts the bit the programer wants and the security signature which is passed to the hardware hypervisor. The OS adds the required values to the signature to make it fit to the job in hand and limit the require resources right down.

Thus a chunk of low level security is there without the programer having to do anything to get it.

Further the smaller the function the tighter the signature can be bound to the function and the easier the signature is to produce.

Also you force not just a standard exception handeling model but also better error mesages and better logging into the system at close to zero extra cost.

This is the bit you get for free doing the bottom up aproach which you tend not to get from the top down aproach.

Don’t get me wrong I like MILS I just think it could do a lot more for the same money with a slightly different underlying architecture.

As an analagy in cars Side Impact Protection Systems,

The top down aproach makes them an expensive addition to an existing design.

The bottom up aproach on a new design builds them in as a structural feature alowing optimal placing etc.

The for free bit, converting the car design to a soft top is oh so easy with non of the old problems where the design just assumed the doors etc where their for the niceties not for safety.

It’s the way you look at things from the outset before even the requirments analysis has pen to paper.

Clive Robinson • March 13, 2010 4:07 AM

@ Nick P,

“I’m mainly a software guy, so this is pretty low level for me. It seems like your DSP is treating the CPU & external stuff as black boxes, watching the IO & keeping track of state. I’m wondering how you will get the CPU & DSP to work together”

The problem I think you are having is you are thinking in terms of the “monolithic slab CPU” we currently have from Intel and AMD etc which has like the “saber toothed tiger” over specialised to do just one evolutionary task well to the detriment of all others.

Go back to a “simpler time” around 20 years ago of PC accelerator cards using multiple RISC CPU’s each with their own private memory block and no need of paging and other Virtual Memory tricks etc.

Each RISC block exists in a “world of it’s own” when running just like an embeded microcontroler. Think about them from the “simple embeded microcontroler” view point of CPU control bus, memory address/data bus and I/O bus (look at the original ix86 CISC design when x was less than 4).

That is the RISC CPU comes out of reset and executes it’s “embeded program” it reads data from an I/O bus “in” device processes it and writes it to an I/O bus “out” device. If the I/O bus is held the CPU halts/pauses (stalls on I/O).

If you think on it a little further it is a hardware representation of a simple Unix Command Line piplined process.

[The C-CPU has no knowledge of security that is done by the H-CPU and system “OS” which sets up the “plumbing” of IPC via “pipes” or “sockets”.]

Now imagine a high end program written as a script that is split into all it’s sub functions. All the sub functions are “pipelined” together by the system “OS”. In simple cases this would be less “efficient” than a monolthic code block but only in CPU “grunt”.

From the “embeded microcontroler” view point, each sub function is the “embeded program” in an “embeded controler” all sitting on a backplane with multiplexed I/O and control buses (as seen in high end Cray and Sun Starfire boxes and IBM Z servers).

This is a proven method of doing “parellel processing” which due to the proliferation of COTS has been virtualy eclipsed.

The addition I’ve made is the hardware hypervisor doing it’s “DSP” on the “function signature”.

So yes “when running a function” the compute engine (C-CPU) is seen as “black box” from the outside, and from the inside “it’s own little universe”. However the “system” does not get to see or talk to the C-CPU engine (CPU+Memory+I/O) it talks to the Hypervisor (H-CPU) that watches the C-CPU when running, and modifies the compute engines memory when the CPU is not running.

The system sees the Hypervisor engine as being on two seperate buses the traditional “data/comms” buses and in addition the security bus (more on these buses and backplanes later)

So the “data flow” in the system passes data to the first hypervisor which passes it down to the compute engine that filters/processes the data and hands it back to the hypervisor which passes it back to the system. The system then takes this processed data and passes it to the next hypervisor in the line.

The only real issue is getting the “embedded function” onto the C-CPU.

Apart from the top control “King/Gov” engine all the other engines have their memory loaded and locked for them (MMU’s and DMA done right are a security plus, more on that later).

Ignore how the top control engine starts up for now just assume it comes up “secure”.

All the other engines are held in reset by their respective hypervisor(s), which in turn are held in reset by the system.

The Gov engine takes control of one of the non IO engines and it’s attendent hypervisor and loads an image into it to act as a simple security schedular for the lower level hypervisor(s).

The Hypervisor as I said is a simplified Harvard architecture and has a code side and a data side. The code side is effectivly “ROM” (see how some DSP chips load code into themselves after reset from a “hidden external EEPROM”) and remains non mutable during the hyporvisor execution. The data side faces it’s respective von Neumann engine(s).

Conceptualy each time a “cell” engine is used for a new function,

1, Both the hypervisor (H-CPU) and compute (C-CPU) engines are reset,
2, The H-CPU engine gets the signiture and bounds loaded on the code side.
3, The C-CPU engine gets the function code DMA’d into it’s restricted memory space via a one way private bus.
4, The C-CPU engine MMU is preloaded and locked.
5, The H-CPU engine is taken out of reset and it takes control of the data I/O to the C-CPU.
6, The H-CPU takes the C-CPU engine out of reset.

And the “black box” is now running a function.

Back to the DMA and MMU and security. Done right it can be made very very secure. It is a question of control and bus isolation.

DMA of the form I’m looking at is quite simple, you have the hypervisor CPU (H-CPU) engine outside of the Compute CPU (C-CPU) engine. The C-CPU has no ability to see anything other than it’s memory and a simple I/O status register. The memory is controled by the DMA controler which is controled by the H-CPU. Thus the C-CPU just sees memory buffers where data “magicaly” apears and disapears and an I/O status register telling it when the magic has happened.

The H-CPU does the “magic” in four ways,

1, Load memory images when C-CPU is held in reset.

2, Load data into “read only input buffer(s)” when C-CPU is “held/paused”.

3, Unload data from “write only output buffer(s)” when C-CPU is “held/paused”.

4, Inspect all memory for “contraband” when C-CPU is “held/paused”.

The MMU is likewise controled by the H-CPU and the C-CPU has no knowledge of it. Thus the H-CPU allocates only sufficient memory and buffer space to the C-CPU to do the function. Thus any low level malware is starved of resources.

The problem with nearly all COST CPU’s is the assumption that the CPU control’s the MMU to do pageing etc it’s self, which means if malware gets “ring 0” it’s game over.

The whole point of this design is the C-CPU does functions of scripted programs and thus does not need to page memory in and out or multi task or any other of those “kernel functions” where low level security holes happen. Thus you can take a simple “off the shelf” FPGA design and hack the design down and providing there are enough pins you can put both the H-CPU and C-CPU on the same FPGA.

If you where going into serious production you can see that with quite a small modern silicon area you could have four “cell CPU” cores each with a private chunk of the 8Mbyte of internal memory and one main hypervisor and four slave hypervisors all in the same footprint a mid to low range Intel/AMD laptop CPU currently has.

Which brings me onto backplanes and data/control and I/O buses.

I/O to the real world is effectivly “untrusted” as it should be and the usual MILS Castle aproach applied.

The low level of the Prison aproach is to leverage “parellel processing” and “scripting” techniques into MILS and other security architectures to get rid of code cutter “glory balls” of programs in a way that,

1, Reduces the granularity to “proven secure functions”.
2, Where the secure functions have low complexity and low resource needs.
3, Where the low complexity alows effective easily tailored security signitures.
4, And the resources can be tightly fitted and controled around the function.
4, Where the tailoring and resource control is done by the build process, not the code cutter.

Thus the low level security against malware is taken almost compleatly out of the code cutters hands. Which means you get the low level security at the cost of system “grunt” but this comes cheap by comparison with traing code cutters and doing low level code audits etc etc.

But as you are doing bottom up engineering you get a whole host of freebies.

First off as I said iX86 wher x is above 4 is overly optomised for single chip “specmanship” and is mostly a compleate waste of silicon in multi CPU systems (it is why we are now seeing Intel go down the multi core route). Thus the silicon that can be freed up will provide plenty of gates for the hardware hypervisor, and still have gates spare.

Forcing parellel programing onto code cutters is very good, it makes their product scalable onto SaaS or “cloud” systems much much more efficiently

Forcing the parellel programing to use trusted functions that are effectivly stand alone programes (ie scripting) forces the correct addressing of “exception handaling” and “error messages” onto the code cutters via the build tools. As these are the root cause of well over 60% and as much as 90% of “in the field” problems we would expect some real gains to be made in this area.

There are other gains such as “code reuse” will actualy work, but hey you want the bad news 😉

Firstly there will be increased overhead in terms of hardware and a decresse in efficiency.

However looking at previous systems that have been properly designed to be “reliable” and “scalable” these extra overheads are relativly small (ie a few %). Which means they are very much irrelavent in most cases.

The usual “object orientated” code aproach won’t work. This is not an object issue but a code cutter issue. As we know from experiance objects fit well in user interfaces but just don’t work in transactions (try mapping objects onto an RDBMS properly, if you think pain is pleasure you will be in seventh heaven). The reason is simply the wrong view point.

Any way many “objects” try to hard to be “all things to all men” which frome an efficiency or security view point is just bad news (just compare the C standard library to the C++ standard libraries to see why this can be bad news).

Oh and the system is limited in what “security” comes for free. It obviates the malware attack issue but does not address the issue of “malicious code”. That is if the code cutter deliberatly puts in malicious high level code this system is not going to detect it (otherwise secure programs would just write themselves). But it does limit most of what a malicious code cutter can do to the point where they are squeased out by “formal methods”…

Clive Robinson • March 14, 2010 6:16 AM

@ Nick P,

I’ll try to be quick a tads more indepth on this reply without getting to deep.

“I still have qualms about the signature approach being unable to differentiate between malicious & non-malicious in real-time”

I can understand that as it takes a whole different “world view” from normal (which is why I keep talking about bottom up not top down design).

Think not of “whole monolithic programs” in the normal sense. Nor of programs in scripting languages like Perl. Think of small functions made as micro programs that are scripted via IPC (Unix shell script and CL utilities).

Each function is very small has a minimal number of loops and next to no storage. It is more of a “filter” building block than a “user program”.

The signatures are “heartbeats” either put into the function by the build tools or exist naturaly via the I/O buffer requests. Thus the heartbeat type and number are known to the build tools that make the “skeleton” of the signature. The hypervisor knows aproximatly what order each heartbeat comes and importantly when and what it’s tag should be.

Any change in this expected heartbeat behaviour is not good news and raises a security indicator for further consideration (ie a cell search).

Thus a “do while” loop would have a heartbeat at the begining to say it’s in the loop and what the limits are, a sub heartbeat at each itteration has the loop count and while test input and result. Likewise any break conditions have heartbeats.

Thus any malware would have to know what nearly unique heartbeat had to go next and when, which would require it to be able to analyse the state of the prison “cell” before the valid heartbeat time out. And as the cell is resource constrained…

The heart beats enable not just the signature but also a very good debug/diagnostic for fault tracing and error messages and why any exceptions occur (remember nearly 80% of faults are not code logic faults).

All unexpected exceptions cause an automatic “cell search” and dump of stack and heap to log for analysis by the hypervisor.

“Are you even really analyzing the IO with the monitor or are you just using it to direct, examine the memory of, and reset the compute CPU’s?”

It is both although I’m not having to do to much IO content anaylsis at the moment it’s a work in slow progress (as am I of late ;). Also it may not be required in many cases for subtal reasons.

“In previous discussions, you described it somewhat like an IDS, but now it just seems like a real-time, code-integrity checking engine for compute nodes.”

It’s probably because I’m focusing in on individual bits at a time.

The heartbeats and the I/O checking give you the signatures for IDS, which also give you some of the code-integrity checking. The random “cell search” gives you full code-integrity “Monte Carlo” style.

So there is no gaurenty low level malware cannot get in briefly but it would have to be something quite extrodinary to do much other than be detected quickly.

“A typical platform is millions of lines of code”

It depends if you have a monolithic kernel like linux it currently has some 12million lines of code that go into it.

If you look at the Minix 3 kernel it has around 4 thousand lines of code.

If you look at the bottom of the C Standard library there are only a very small handfull of functions that need kernel support.

Going to an I/O Buffer model strips that down even further think of it conceptually like that of the bottom USB comms layer (which I’m seriously thinking of doing in silicon) in some respects.

In other respects IPC by jumping windows provided by the simple VM MMU controler on a block of external memory (which is like software passing by refrence rather than passing by value, and can have similar issues).

“which probably correspond to hundreds of thousands of functions,”

Err no actually about 200 hundred are all that are required and only a handfull get regularly used.

“each requiring a computer/hypervisor pair.”

Only whilst they are in active use. Remember that like any tasking computer you page out and task switch to make optimal use of resources.

When you staticaly link the functions a lot of the code bloat and security issues vanish. And the function code becomes very small.

Think of the difference between the Unix command line and stream editors “ed” and “sed”, and a wysisyg editor like MS Word. Then look at a point or two in between such as vi or jstar and web based editors such as that that appeared in the likes of GMail. There are many ways of doing the same things and each has it’s advantages and disadvantages. But with a little ingenuity you can do what vi originaly did which was to build on a much simpler primative or function (ed).

“Even the simplest CISC cores take up a significant chunk of any affordable FPGA.”

Yup and CISC is most definatly not the way to go it’s an evolutionary dead end as Intel have discovered.

Some of us are old enough to remember what ARM stands for (Acorn RISC Machines). And also building CPU’s out of early Texas DSP chips to support “Forth” in a significant way which actually only needs 27 base instructions to work well which kind of makes even most RISC CPU’s look bloated.

You do however have to add a few other hardware instructions for Maths and buffers but it’s an easy tradeoff senario in most cases.

[Forth for those that are not familiar with it is a “stack based” RPN language which has been known to beat optomised C code hands down. The RPN and stacks make analysing it very easy and light work.]

Just to show how crazy it can be sombody developed a LISP system on a Forth engine the performance results where at the time eyebrow raising to say the least.

“Custom designing functions & acknowledging they will probably be multiplexed to reduce compute node requirements leads me to believe the system still needs almost a hundred thousand FPGA’s or tens of thousands of high end FPGA’s.”

Err no the functions are software images just like any normal program. The Kernel code is very minimal and modular and the C-CPU very “light weight RISC” in most cases (and not “pipe-lined” which causes all sorts of efficiency issues).

Function code is compiled into static “micro-images” (as are code moduals in embeded systems) and is loaded very rapidly into the C-CPU memory space through the H-CPU.

[There are ways where you can bank switch the memory and registers to the C-CPU so “context switching” is just two or three clock cycles, such are the joys of stack based designs.]

IPC / data comms is done currently by memory mapped buffers into a large shared external memory space via a simple VM MMU controler. Thus the results in the output buffer of one “function” become the input buffer of the next function.

Importantly from the security context neither the H-CPU or C-CPU have access to the control side of the VM MMU. Thus they have absolutly no knowledge of the external memory just a view of one or more buffers in a fixed place in their memory map.

[This is done by the externaly controled simple VM MMU controler the C-CPU simply changes a status bit to say it’s finished with a buffer and halts on the external system updating the simple VM MMU controler to the new external memory window and clearing the status bit. This can usually be done as a page counter with address offset so is very very fast (just a couple of clock cycles).]

Imagine the IPC as a circular buffer of buffers with multiple functions using the one circular buffer of buffers in “lockstep”.

Thus imagine the functions lined up (A,B,C,D,E,F) each with one “in” and one “out” buffer. B’s out buffer will be switched to become C’s in buffer likewise C’s out buffer will be switched to become D’s in buffer. Thus C will stall/block if B has not finished writing to it’s out buffer, preventing it being switched to become C’s next in buffer.

However A’s in buffer gets fed from a “file” and F’s out gets fed to a “file”. These “files” are again just buffers to either large memory buffers for HD files or heads of streams.

“Even assuming one could do a lot with an Intel-type wafer”

Ahh my mistake I was using Intel chips mainly as a “silicon area/size” comparison, not a direct function comparison.

That is to give an indicator of gate count pin count internal memory potential and cost that can be achived. Thus alowing a mental comparison to how many of these little beasts can be put in a package (and might well have done if the IBM PC team had picked the Motorola 68K instead of the Intel 8086).

I guess I should have looked for a high end Renedering GPU Chip that would be closer but in most cases their internal structure silicon area is proprietry knowledge so it’s not easy to use as a comparitor for what’s possible.

“Intel’s manufacturing processes are far ahead of the curve compared to FPGA’s, proprietary, and unlikely to be released to give them competition.”

Yes, but unlike GPU manufactures they give away a lot of data in terms of gate density structures and chip area speed through put etc etc in marketing blurb. Which is why if can be used as a fairly reliable indicator of what is possible.

I’m using FPGA’s simply for prototyping hardware, in the same way I used to use reprogramable ROM as state machines and in one case an RF IF (but that’s another story altogether).

If you have a look around you will find you can get PCI cards with multiple FPGA’s on which means you don’t have to even think about picking up a soldering iron 8)

“They’re too invested in… …Looking at… …NVIDIA parallel chips, I don’t see anyone who can sqeeze enough multiplexed functions on a chip to make this design cost-effective.”

Look at what Sun did with their RISC CPU and Cray designed multiplexing (Starfire systems such as the 10K), likewise what IBM do in their Z machines.

They are still (maybe in Sun’s case) doing it and producing throughput Intel still has wet dreams about. But Intel can not achive because they have gone to far down an evolutionary culdersac of single chip CISC. Then to get performance Intel have had to add all the pipelining, lookahead and hyperthreading etc etc which chews up a lot of silicon for very little benifit.

Intel chips are designed to “appear to be all things to all systems”. But in reality they do a limited set of some things very well for “office desktops”. And thus perform badly in high end single user systems and low end servers and just cannot hack it in high end servers (look at context switching times and the underlying reasons).

But the Intel chips are extrodinarily cheap for “office desktops” because the overly expensive designs are amortised across multiple millions of chips.

The fact that they do baddly in high end high integration servers is the reason we have gone to clustering. However even here people are looking at using games consoles and graphics cards because the GPU’s have way way more performance for properly parellel tasks, not just in silicon area but in bang/watt and bang/chip cost.

Go have a look at some early ARM processors and have a look how Unix did the IPC plumbing etc and have a think on how a simple VM MMU controler can be added on to map “buffers” for doing all the I/O above the hardware layer.

What I have done is move the tipping point by taking a new look from the bottom up. Instead of using desktop mother boards for making clusters I’ve used FPGA’s on PCI cards.

I’m also currently playing (all be it very slowly) with PC104 cards to see if I can migrate up the COTS stack to get similar effects for considerably less cost. There is a reason for this and it is to do with “multi media” some functionality will not go into such limited resources as FPGA’s. And like it or not high integration PC motherboards do make cheap thin clients for user I/O work.