Schneier on Security

Nick P • November 22, 2014 2:00 PM

Modern virtual desktops are far from secure: A historical perspective & case study
(inspired by this news article)

There’s little new or Matrixy about this stuff. In a nutshell, a privileged mode or component was added to the processor without security in mind, it was attacked, and the system was compromised. This is the M.O. of INFOSEC going back to original MULTICS security evaluation [1]. It showed the many layers from hardware to apps that could be attacked, subverted, or critically fail on their own. This eventually led IBM in 1976 [2] to apply such lessons to their VM/370 hypervisor product. The resulting product split the system into virtual machines running on a hypervisor, KVM/370, with a security kernel embedded. Even before that, you could run the VM hypervisor on top of the VM hypervisor. (“Matrix shit” in… the early 1970’s??? Wait, where’s the novelty?)

This didn’t stop. Karger, one of the MULTICS evaluators, designed in the early 1980’s a secure VMM [3] to virtualize “VAXen” for users of different clearances. Their A1 security engineering techniques included layered design, minimally complex components, mathematical proofs of correctness, pentesting, processor microcode modifications, and other established techniques for increasing whole system security. That the system was usable is evident because they developed it on itself & got positive mention from beta customers. It was canceled for business reasons such as time-to-market issues. Technologically, the KVM/370 and VAX VMM set the baseline for how one should begin securing a virtualized system. Most modern work doesn’t even meet that baseline, although newly discovered attacks and issues demand an even stronger baseline.

Later on, the development of GUI’s, desktops, and many UNIX’s led to Compartmented Mode Workstations [4]. Many of them took a tiny security kernel at the lowest layer, some trusted OS-style software at a higher layer, made modifications to support isolating different security levels, made it more user friendly, leveraged hardware rings/segments for extra protection, and then deployed that in the market. Trusted Xenix is an example which also added immunity to some issues (eg Setuid vulnerabilities), covert channel analysis, & more NSA pentesting. Best designs, although not GUI’s,broke OS into components running as separate domains with kernel enforcement. Examples were XTS-400 and KeyKOS. Modern CMW’s like Argus Pitbull unfortunately leverage monolithic kernels. At least Argus warns of that risk.

So, fast forward a decade or two to Invisible Things Lab. The team demonstrates an actual attack that leverages SMM risks described in mid-90’s papers on x86 security. They also show privileged layers in a system with no security applied can be attacked. This kind of obvious stuff somehow makes headlines in IT media despite being ancient, sometimes totally solved, problems in published INFOSEC research. They intended to build something better. At the time, state of the art in academia was a mathematically verified isolation processor (AAMP7G), several secure microkernels (eg EROS), several trusted GUI’s (eg Nitpicker), and several implementations of user-mode Linux on microkernels with one having device driver wrappers for compatibility (eg OK Linux). INTEGRITY Desktop, commercially deployed in (2002?), had already put virtualized Windows, Linux, and native-on-kernel apps all in the same system running on a microkernel with isolated networking, robust transfer between VM’s, processor security extension support, etc.

QubesOS project begins. They start with the Xen virtualization platform. It has a large TCB in Dom0 which depends on Linux code, which has a bad security track record. They then added a subset of CMW features (eg labels & colors) with a virtualization approach similar to the INTEGRITY Desktop and Dresden work. They then tried to make their VM launching very fast like Dresden had already done (sub 1s on cheap hardware). They wisely put the networking and firewall stacks in a separate domain like microkernel & MILS kernels do. They’ve probably added plenty more since I last looked at it. The resulting system is quite easy to use, supports legacy hardware, will limit vanilla malware that isn’t targeted at them, might stop some malware targeted at them, and has other benefits of virtualization.

So, what of the security? The Five Eyes, Israel, Russia, and China have all shown expertise at hitting every layer of a system. I describe a bunch in my own framework [5] I used to guide my security engineering and started sharing publicly as a response to Snowden leaks. Sophisticated attackers hit every one of these layers. Hence, they must be developed or protected with extremely rigorous development processes: Orange Book B3/A1, Common Criteria EAL6/7, or something similar. Empirical evidence going back decades shows anything less produces software full of 0-days waiting to be discovered along with insidious covert channels. Systems designed with the high assurance processes at least survived strong pentesting at the time and a few like Boeing SNS Server have gone unbroken (far as anyone can tell) for decades.

Let’s look at Qubes in terms of what real security takes. Qubes runs on blackbox microcode, firmware, and so on partly developed in countries with either active espionage or that can compel backdoors in secret. Subversion should be easier than some architectures not tied to such countries. The Linux-derived Dom0 is a risk factor given that both black hats and nation states regularly find Linux 0-days. Neither the code nor the development tools they use are developed with high assurance techniques: mostly low to medium. There’s also been little to no independent evaluation by qualified, whole-system pentesters. (Their team is talented at review, though.) The covert channels I predicted, due to no mitigation by about any VMM vendor, finally turned up in academic analysis of the underlying Xen platform. If that sounds minor, let me rephrase it: IT security field just recently noticed a problem in VMM’s that was in official certification criteria and most landmark INFOSEC papers going back 30 years. This happens depressingly often in both commercial and FOSS INFOSEC development…

All in all, another failure to learn from the past. The project doesn’t apply the critical lessons learned from security design/evaluation of previous work: MULTICS, IBM’s KVM/370, VAX VMM, CMW’s, microkernels, MILS separation kernels, etc. The TCB is overly complex, not rigorously developed, lacks POLA, leverages black box firmware of unknown quality, and runs on a possibly subverted processor. The system should be considered insecure against medium to high strength attackers until the problems are removed. It’s doubtful that the problems can be removed if they continue to leverage Xen, vanilla development libraries, and support as much COTS hardware as possible. A shortcut that might knock out some issues is porting it to something like Cambrige’s CHERI processor, which already runs FreeBSD with Capsicum. Many code injection issues might be knocked out at Xen & above immediately, while stuff below can be reduced or eliminated during an ASIC conversion.

I might still use it for desktops that don’t contain assets of significant value. The QubesOS team did great work on usability, performance, and compatibility with a reasonable amount of COTS hardware. VM’s also make for easy recovery if problems occur. I might also use it for rapid development & testing of special purpose, low TCB VM’s that can be ported to higher security architectures later. It’s just not suitable for protecting my data or system from the kinds of attackers that matter. I can’t recall off the top of my head any FOSS solution that is without a lot of custom work. That won’t change until developers of security-critical systems learn the lessons of the past and actually apply them. I also encourage FOSS types to improve on them like some academics and commercial firms are already doing.

Note: This is not to single out QubeOS. Security-wise, they’re doing much better than many mainstream virtualization packages. There are others doing better than them. It all varies project by project, year by year. The critique is of the common denominator: none of them can be secure due to lack of highly assured secure development process with right requirements in place. Additionally, better techniques have been around and aren’t used. Yet, in the linked article, the writer was mystified by a low assurance rehash of old technology and thought that would stop blackhats (including NSA). That’s a very bad, common mistake I aim to prevent with writing like this.

[1] http://www.acsac.org/2002/papers/classic-multics-orig.pdf

[2] http://www.dtic.mil/get-tr-doc/pdf?AD=ADA109316

[3] http://www.cs.dartmouth.edu/~ccpalmer/classes/cs55/Content/resources/vax_vmm.pdf

[4] http://web.ornl.gov/~jar/doecmw.pdf

[5] https://www.schneier.com/blog/archives/2013/01/essay_on_fbi-ma.html#c1102869

Nick P • November 22, 2014 3:52 PM

Update on research for 2014 in secure virtualization. Some good finds.

They’re encouraging people to give them feedback. Anyone have practical suggestions that the target audience (businesses/government) would apply go ahead and tell them.

Nick P • November 22, 2014 9:55 PM

@ Justin

Apology accepted. I have my moments too. 😉

re virtualization

There are a number of problems that virtualization can be a useful tool in solving. Here’s a benefit sheet to start with:

http://www.infoworld.com/article/2621446/server-virtualization/top-10-benefits-of-server-virtualization.html

And remember that, although some custom OS’s might not need it, we largely use virtualization for the likes of Windows and UNIX/Linux that have “needs clean slate redesign but we’re stuck with it” written all over them. Two of the first uses of virtualization were improved resource allocation and legacy.

Legacy. So, you have to run apps that are tied to a particular OS, configuration, or hardware. You have little or no control over that aspect. If you bought it a while back, these might not be made anymore. A virtual machine on modern hardware lets you run your beloved or mission-critical apps without any need for source code, OS, or even their hardware. That software just needs to be designed once and maintained on modern hardware. This makes it cheaper than rewritting every app. Also lower risk of breaking stuff. Real-world examples include old mainframe apps (eg VM/370), modern “Amiga compatibles,” Charon’s VAX/Alpha VM’s, DOS emulators in Windows, and even wrapping up apps tied to old versions of Windows on a current version. I’ve seen all in practice & a business case for it.

Resource allocation. OS’s have efficiency issues. They might stall the system waiting for I/O, have a poorly designed scheduler, not be so great at memory management, etc. Putting any or all of that in control of a VMM might let you give (or reclaim) unused physical resources to those instances that need it most. This is how virtualization can sometimes improve utilization, giving you more hardware ROI and/or performance.

Isolation. You can put an entire system into a file or piece of physical memory. You can install a bunch of untrustworthy (insecure or unreliable) stuff into it. You can run it without serious problems spilling outside the box. This is great for testing 3rd party code, apps, or especially OS’s. A secure virtualization system can pull this off in face of malicious apps targeting the system. Hence a need for those.

Testing. This builds on isolation but there’s more to it. A virtual machine can let you run arbitrary code on an arbitrary configuration. You might have a bunch of different instances representing different machines. You can load them, change them, inspect effects of running code, revert to clean slate, dispose of them, etc without messing with a single piece of extra hardware. These days, even hardware circuits are often simulated in chips run in software or on FPGA’s to reduce cost of designing them. The most common example, though, is testing a new version of an app for N different machine setups… using one machine. You can even have a test running in a VM instance while you compile other things or automate them with scripts.

Uptime. Offerings like mainframes, NonStop, clusters, etc already give us plenty in this. The use of virtualization allows us to use normal OS’s that may or may not be designed for uptime*. Not designed for it, yet with other compelling reasons to use them over those more expensive solutions. The virtualization solutions can do a number of things to keep sites running, including live migration. That’s cool shit, too.

• “Five 9’s FreeDOS VM’s available for $1,000/year. Anyone? Anyone? Bueller?”

Faster server provisioning. This was in my response on the Qubes article. Qubes and Dresden’s demos demonstrate instant creation of new machines. This naturally has deployment advantages if you set the system up on a few large machines with plenty of resources and you’re only charged for what you use. That’s the mainframe model. I’ll add that you get improved security/efficiency if you extend it to be able to drop unneeded privileges after startup and also selectively include code. That it, the same base code is already in RAM for all to draw on but each machine has a configuration profile with the minimum it needs to do its job. And that is set to be unchangeable before it starts running its production app.

Data center footprint and energy savings. Remember that’s there’s a lot of stuff on the board besides CPU, memory, and I/O. And utilization often sucks for a regular server OS. So, the resource allocation and consolidation benefits that get you more out of the system also reduce financial or environmental waste tied to that system.

Desktop virtualization. This is a complex field with a variety of benefits and gotchas that kill many projects. The benefits include most of the ones you see above except with additional benefit of centralized control over something usually centralized. The thin clients are also easier to lock down. One success story I read involved moving over 200 desktops to one Bull mainframe (with extra for testing) and they reported things performed well. Virtualization schemes are basically emulated mainframes so they either can or eventually will be able to reliably offer this capability. Biggest security benefit outside centralization & utilization is automated deployment of patches to all users without messing with their individual machines. Even many individualized machines might be scripted modifications of a core image and this benefit still applies.

Multiple untrusted parties. This builds on isolation and adds security primitives along the lines of MLS/MSL/MILS. The idea is that shared resources are controlled to prevent leaks of information or attacks through them. The VM’s themselves are also controlled in terms of resource use and capabilities. The result is that the same machine can process stuff with different classification levels, belonging to different companies, and so on. The military would love this as it would save them a lot of hardware.

Ephemeral computing. A virtualization system’s isolation and resource management properties can be used to ensure it knows where application data is. It might even encrypt that data as it enters or leaves the processor as numerous research prototypes have done. The entire systems contents are connected to one or more keys in the TCB. The virtualization system can wipe a partition or the entire system in micro to milliseconds. Residual information from released memory is also encrypted as such that a VM acquiring it has no leaked secrets even if it wasn’t overwritten. There’s numerous benefits to this sort of thing.

So, there’s numerous benefits of virtualization that you get without changing the code running in it. Changing the code, as in VM-aware apps or paravirtualization, can get you even more benefits like isolated I/O or accelerated computation even in OS’s or apps that don’t natively support it. The last thing we want, though, is to try to get these benefits and have those evil hackers “pown” us in the process. So, if virtualization is beneficial, then securing it makes a hell of a lot of sense.

And there’s a lot of good work in that direction.

Nick P • November 23, 2014 8:06 PM

@ Chris

“But, in terms of having everything run in a separate VM, how would this be better/different than having multiple OS/partitions on a single machine, each using FDE, making it more difficult for malware on each to affect the other (aside from boot sector issues, maybe you could use a secure boot type scheme or have the bios run a hash test on it before it loads)?”

If VM’s instead of partitions, you can run multiple VM’s simultaneously on the same machine, do copies/backups as easily as moving files, have multiple versions of the same apps without DLL hell, and even run the same system on other people’s PC’s so long as you install a VM app on it. There’s also the benefit that OS’s or software not supporting your hardware might be able to run in the VM.

@ Thoth, Grauhut

The seL4 development also came with a user-mode Linux to run on top of it. Most of the L4 family has paravirtualized Linux layers. Verifying that the isolation properties apply to arbitrary inputs from the Linux layers is a shortcut to a near EAL7 virtualization system. Look into Nova microhypervisor as well because it’s designed for VM’s, tries to leverage formal methods for correctness (not necessarily security), and is open source.

Additionally, the LynxSecure commercial hypervisor leverages Intel VT for full virtualization and paravirtualization with Linux. They even went with my suggestion to port their DO-178B OS to run on their hypervisor. Secure hypervisor layer and at least a safe, resource-controlled OS layer for apps. VxWorks MILS kernel does the same thing.

And, Grauhut, thanks for the L4/KVM paper! I can’t believe missed it given all the Dresden papers I’ve posted and referenced. I figure I saw KVM and got scared away. Looking at it now, it’s amazing they added it with only 100loc worth of modifications.

@ Jason, Apples

Appreciate it. 🙂

@ Subito, cepha

The proper use of virtualization involves splitting your activities between sensitive and risky stuff. The risky stuff is in a VM. Your sensitive stuff doesn’t go in it. Risky stuff VM getting compromised doesn’t compromise your system as a whole and you can restore it to clean slate easily. You can actually do that regularly for either system just in case. Other measures or tools are used to try to prevent attacks from hitting you. An updated Firefox with HTTPS Everywhere, NoScript, and Sandboxie is pretty good by itself, for instance. In a VM or VM tied to a LiveCD, even safer.

@ Justin

Theo de Raadt will argue with anyone who doesn’t like it his way. He hates virtualization, mandatory access controls, etc. Let’s leave him aside except for his point that it brings in an extra layer that can fail or be attacked. And people that can’t get OS’s right will probably not get it right. I agree with those points, yet the solutions are obvious: make that layer a lot simpler that what’s above it and code it well. Nova or an L4Linux would be good examples.

x86 is actually kind of a separate issue. It really is crap from a virtualization standpoint, yet that didn’t stop VMware from implementing all sorts of functionality. Intel realized the problem and implemented Intel VT among other features as a solution. Leveraging that rather than vanilla x86 is possibly the best approach. Hypervisors like LynxSecure and Nova microhypervisor do that with a strong focus on simplification and code correctness.

Another option, taken long ago by Transmeta and recently by Loongson, is to emulate x86 at the hardware level. There’s quite a performance hit to doing this so it’s not for performance critical apps. However, it can be useful for legacy and one can even build in advanced security techniques at the hardware level to spot likely attacks (eg stack pointer overwritten). The cool thing is the chip might be a better-than-x86 chip by default whose microcode is changable. That lets it be used as RISC with custom instructions, an emulator or something else.

@ Nate

” Virtualisation has always seemed strange to me. If you have to virtualise an entire OS (and we do), isn’t it basically an admission that the OS itself has utterly failed in its prime mission of providing virtualised access to the hardware and separating the processes on the machine? That’s what an OS’s only job is, and it can’t even do that properly. ”

That’s not really an OS’s job. An OS’s job in the marketplace (or even FOSS popularity contest) is that it does what users want. This might be run certain apps, have certain features, support some standard, run on certain hardware, and so on. So, OS’s evolve a feature set that meets the demands of their users. (Hopefully…) Demand for personal computers was for them to constantly be faster, cheaper, prettier, and support more personal applications. So, this became what they were good at. Similar for servers, except more focused on running business applications, supporting standards (esp Internet), and costing way less than mainframes or enterprise RISC systems. Note how the kinds of things virtualization do don’t really fit these trends at all until hardware got cheap enough for it to fit servers and corporate desktops…. sometimes.

Virtualization was originally invented as part of the effort to make time sharing practical on hardware mainly used for batch processing. The CP/CMS system on IBM System/360 virtualized all the hardware of the system. Each user running the VM could have their own custom system without changing out the hardware. Just think having an account on a LAMP web host vs having a virtual private server. You get more control, more flexibility, possibly better performance with your custom OS, no extra cost of the machine (charged for use), and resources are still controlled by an underlying software. Mainframe OS’s supported some of these benefits, but not those that required changing the OS itself.

That team even created the System/370 version of their product, then called VM/370, by simulating a System/370 on a System/360 using their virtualization tool. That you can customize everything from hardware to OS to software with a virtualization tool comes in handy in many use cases. VM alsi ended up being one of the main ways IBM maintained backward compatibility with arbitrary legacy apps and OS’s on later hardware. Proves virtualization is great for ROI: write once, keep running them. There are apps from the 1960’s still doing their jobs in some companies. And now we’re seeing Windows and NIX servers being done the same way decades later.

Back to your point, though, on a better design. This is totally true and there were better designs. Most were prototypes in academia or failed commercial products. One thing they had in common was to create a standard representation for implementing CPU, memory, I/O, and protection mechanisms. These need to be easy enough for hardware to support and flexible enough to do a lot of stuff. Then, you separate their use from how they’re managed and push the how to upper layers in their own components. The capability-based systems probably did better than most at achieving this. System/38 -> AS/400 -> IBM i shows they can occasionally make it in the market and for a long time. I’d look into those. (Esp look into KeyKOS.)

Even still, IBM added virtualization (PowerVM) to the i Series. The reason was for running legacy and mainstream OS’s side-by-side on same machine. Mainstream OS’s whose source they mostly don’t control. And this is why virtualization is relevant sometimes even if you have a system that wouldn’t need it for any feature you want. The responsibility for the majority of a codebase can be shifted to others while you put a box around it for resource controls and fault/malware containment. Virtualization isn’t going away.

So, the best approach is to simply integrate it with legacy technology and make the kernel/VMM/whatever quite secure. The L4 family led the way, esp OKL4, by creating a kernel that can run native apps on small TCB, do device drivers, borrow them from legacy OS’s in VM’s, run VM’s, and so on. Quite flexible. Look into OKL4, OK Linux, Device Driver Environment (I think that’s the name), etc. Look at DROPS OS and Nizza work, too. You’ll see how you can have the same hardware and core OS/kernel underneath while being flexible enough to do almost anything.

re one language

LISP and 4GL’s already did what you describe for most of that. The trick is to have strong metaprogramming to allow DSL’s to be built (LISP) or build DSL’s on top of a very HLL (4GL). The 4GL’s were also much easier to use and more readable on the stuff they were meant to do. Haskell DSL’s represents an evolution of them where the advantages of functional programming and DSL’s can merge. This handles correctness, conciseness, amenability to formal analysis, and concurrency it seems. More research into improving its performance and assessing its security issues is needed. Yet, if you’re looking for ideal, it’s going to be something like 4GL’s that’s better than Haskell or LISP at integrating diverse ones. Or an improved Haskell or LISP. 😉

EDIT: I just read Justin’s and your follow up posts. It seems you figured it out before I got you an answer. Least that confirms my thinking is more likely to be on the right track.

Nick P • November 23, 2014 9:15 PM

@ Chris Abbott

Long term solution is difficult because of the demand for backwards compatibility. If we keep standards, but disregard compatibility, then we just create clean slate secure hardware and software systems. If we stay compatible, then we’re looking at containment, detection, recovery, etc for stuff that’s going to get hit. That’s the default right now. Virtualization, especially if strengthened, is a cheat right now to at least try to keep that mess in one box and maybe also do other things on another box. That other box might even be clean slate.

The current, mainstream virtualization tools have had all kinds of flaws. Malware authors will shred them if they become popular. Actually, many security researchers are doing exactly that due to widespread deployment of some in cloud applications. The good news is there are commercial, academic, and open source efforts that I mentioned above that are trying to make strong, easily evaluated hypervisors. The Genode OS is an example of a clean slate design with more secure native apps, support for legacy ones, and supporting numerous kernels. One it supports is the Nova microhypervisor.