Automatic Disk Compression Stirring Debate Among Users

By Bruce Schneier
September 6, 1993

Automatic disk compression programs promise to effectively transparently double the storage space on your hard drive. Although most can live up to this claim, several factors, including performance and reliability problems, have sullied the images of these products.

Given the prevalence of cheap disk drives, some doubt the wisdom of using these products at all. Many users report no problems, however, and have come to rely on automatic disk compression to squeeze every last byte out of their storage devices.

There are two types of background compression products: file-level and driver-level. File-level compression programs compress files during idle periods and uncompress previously compressed files as they are read from disk. Driver-level compression programs replace or supplement a drive's SCSI controller and automatically compress all files as they are written and uncompress them as they are read.

Each compression method has advantages and disadvantages in terms of speed, price, compatibility and compression efficiency. Choosing among the products depends on many factors.

File-level compression

File-level compression programs include Aladdin Systems Inc.'s StuffIt SpaceSaver, Alysis Software Corp.'s More Disk Space, Fifth Generation Systems Inc.'s AutoDoubler and Now Software Inc.'s Now Compress.

Proponents of file-level compression cite its speed and safety. "I used AutoDoubler on my IIci for over a year," said Gary Gray, research fellow in the engineering mechanics and astronautics department at the University of Wisconsin at Madison. "I never noticed it was there, never lost any data and never noticed a slowdown." Gray said that AutoDoubler typically compresses PostScript files from one-quarter to one-third of their original size.

"[File-level compression] slows things down considerably if you run the disk really full," said Dave Sholty, producer for On the Spot Productions, an advertising company in Chicago. "Otherwise, it is transparent and automatic."

Unlike driver-level compression programs, file-level compression programs let users compress and decompress files manually as well as automatically. File-level compressors also allow users to selectively turn off compression for certain files. This is important for frequently modified files, such as Adobe Photoshop swap files and certain applications.

"Some applications, such as America Online, just don't like to be compressed," said Norman Murray, computer consultant for the biology department at the University of Rochester in Rochester, N.Y. "Others, such as Microsoft Word, are insufferably slow when compressed."

Users can avoid this problem either by not compressing applications or by dedicating enough RAM to the file-level compressor so that it is able to load the entire application into memory.

Since file-level compression programs interact directly with the system, they are more prone to extension conflicts than driver-level compressors. Jim Owens, chemist at the National Institutes of Health in Bethesda, Md., used AutoDoubler for months without a problem. "Then I had trouble opening files; the creating application could not be found," he said. Expanding the files did not fix the problems, Owens said. After several attempts to fix the problem, he gave up, reformatted his drive and reinstalled everything except AutoDoubler, without further problems. "I cannot be sure the fault is in AutoDoubler, but I am afraid to use it again," he said.

William Leininger, software engineer for Green Dragon Creations Inc. of Lake in the Hills, Ill., said, "File-level compression violates certain assumptions you should be able to make about your disk structure and can cause problems with other INITs."

Driver-level compression

Driver-level compression products include Golden Triangle Computers Inc.'s TimesTwo, Alysis' eDisk and Stac Electronics' Stacker for Macintosh. Proponents of driver-level compression consider it a significant improvement over file-level compression.

"I like everything I've heard about driver-level compression," said Ben Liberman, a computer consultant in Chicago. "It's transparent to INITs; everything just sees a larger drive. Driver-level compression programs change the way a drive is treated instead of compressing individual files."

Driver-level compressors are generally slower than file-level compressors because they must uncompress and compress whenever they read or write data; however, this depends on several factors.

"If the hard disk is faster than the processor, [driver-level compression] will slow things down a bit," said Robert Altizer, software quality assurance manager for Motorola Inc.'s ASIC division in Chandler, Ariz. "Stacker actually sped up my home machine because the processor was much faster than the hard disk."

Timothy Ellis, senior member of technical staff at GTE Government Systems of Waltham, Mass., cannot detect much speed difference between running with Stacker and running without it. "Opening and copying the typical file is identical. I effectively have twice my disk with very little pain," he said. Ellis reported one compatibility problem; he could no longer use his DiskLock security program.

Many users have tried both file-level and driver-level compression and report different reasons for preferring one method over the other.

"I switched from AutoDoubler to TimesTwo. I thought I would gain some room, since [TimesTwo would also compress] my large system folder," said Pascal Gosselin, Apple support specialist at Omer DeSerres Informatique of Montreal. He ended up with less available room. "I'm going back to AutoDoubler; TimesTwo slows things down too much," Gosselin said.

The fact that driver-level compression programs alter the hard disk structure makes some users nervous. Once on your hard disk, a driver-level compression product can be difficult to remove. TimesTwo cannot be removed without first backing up and then reformatting the hard drive.

"I have not considered driver-level compression because I am not willing to suffer the performance hit of a driverwide level," said Igor Livshits, Macintosh support specialist for the National Center for Supercomputing Applications at the University of Illinois at Champaign-Urbana. "Also, I do not trust the technology and am attached to the higher level of control file-based compression offers."

Rick Zeman, a free-lance writer in Laurel, Md., said: "I don't think it has evolved far enough in the Mac world for me to trust my data to it. Anything that makes it tougher to get the data off the hard drive in emergencies, I steer clear of."

No compression

Dave Nebinger, biology network manager at Bowling Green (Ohio) State University used a file-level compression product for a while before removing it. "I got tired of being inside an application, getting the standard file dialog, and not seeing any of the files that had been compressed automatically or manually," he said. "It was more trouble than it was worth."

Stan Twiefel, computer lab technician/supervisor at the University of Texas in Austin, said, "I've had a few problems with [automatic compression products] in the past, and am very leery of them." Once, Twiefel lost the entire contents of his hard drive when the file Stacker created to maintain the integrity of the compressed data was corrupted. "The integrity of my files is worth too much to trust them to a compression program."

Others feel that the high speeds and low costs of storage devices today make file compression unnecessary. "You spend all your money to get a fast computer and a fast hard drive, and then you slow it down by putting something in the middle," Liberman said. "You should buy a larger drive if you really need the space."

File-level compression vs. driver-level compression

Automatic file-level compression products are extensions that compress files in the background after the Mac operating system saves files to disk normally. File-level compressors search for uncompressed files when the computer is idle, compress and save them, and delete the uncompressed version. When you open a file, a file-level compressor either uncompresses the data so it's in its native format or loads it into memory. In the event of a disk crash, the original file is preserved.

Driver-level compressors replace or supplement a drive's SCSI driver. They compress files as they are written to disk and uncompress them as they are read. Because they work directly with the SCSI controller rather than the operating system, driver-level compressors are less likely to conflict with system extensions. Since all compression takes place when a file is saved, driver-level compression is generally slower than file-level.

Disks are divided into equal-size allocation blocks that range in size from 0.5 Kbytes to 32 Kbytes, depending on the capacity of the disk (the maximum number of allocation blocks is 64,000). Mac SCSI drivers cannot store more than one file in a single block - if a file is smaller than an allocation block, the extra space is wasted.

File-level compression programs do not compress files in the System folder or any files that are smaller than the allocation block.

Driver-level compressors modify the block structure on the hard disk. This enables them to store data from more than one file in a single allocation block, so they do not waste space like file-level compression products. In the event of a crash, however, this nonstandard block structure can interfere with disk-recovery tools.

There are differences between the ways driver-level compressors interact with the SCSI controller. Golden Triangle Computers Inc.'s TimesTwo replaces the disk driver entirely; third-party drivers such as Casa Blanca Works Inc.'s Drive7, La Cie Ltd.'s Silverlining, and FWB Inc.'s Hard Disk ToolKit are incompatible with TimesTwo.

Because TimesTwo replaces the driver, users can lose specialized disk driver features, such as automatic bad block recognition. Also, TimesTwo does not recognize and will overwrite existing partitions on hard drives. Once installed, TimesTwo can be removed from a drive only by backing up the data and reformatting the drive. Golden Triangle has to make sure Times-Two is compatible with the various types of SCSI hardware.

Stac Electronics' Stacker and Alysis Software Corp.'s eDisk work with the existing SCSI driver; therefore, they let the user take advantage of specialized driver features and can recognize partitions. Alysis and Stac Electronics must test their software with third-party drivers to ensure compatibility.

earlier essay: Emergency Recovery Tools: Raising Data from the Dead
later essay: Mac Development Tools Get With the Program
categories: Non-Security Articles
back to Essays and Op Eds

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc..