I’ve already written about the guy who got a new trial because a virus ate his court records. Here’s someone who will have to redo his thesis research because someone stole his only copy of the data. Remember the rule: no one ever wants backups, but everyone always wants restores.
I have no idea if that image is real or not, but I’ve been hearing such stories for at least two decades.
Tanner • April 30, 2013 1:46 PM
There are two types of computer users: Those who’ve lost their data, and those who will.
Geoffrey Kidd • April 30, 2013 1:46 PM
At least his field is chemistry and not computer science.
bph • April 30, 2013 1:47 PM
A colleague had this happen, someone stole his desktop out of his office with his only copy of his thesis data.
He actually got the disk back, very very lucky.
Brett • April 30, 2013 1:53 PM
Backup, backup, backup… This isn’t just about computers. Ask everyone you know with a digital camera if they backup the pictures from the camera. My guess (based on people I know) is more than 50% never back them up. They stay on the card and maybe they buy a new one, maybe they just upload them walgreens or facebook or what ever.
Me, I have all of our pictures (since December of 2000) backup up in three places on two kinds of media. No, I’m not paranoid about loosing them, I just know if I do my wife would divorce me.
kingsnake • April 30, 2013 2:08 PM
Brett: With most guys, if their wife found the pictures they would get divorced …
Joe D • April 30, 2013 2:11 PM
This was my takeaway from that Matt Honan thing from a few months back.
He only had one copy of all that stuff out on the cloud, assuming that the cloud would be his backup.
But if there’s only one copy, even if that copy is out on the cloud, you have no backups.
Brian B • April 30, 2013 2:42 PM
Steven: one things is for sure. 100% of the restores will fail if there are no backups.
That said, there are many solutions out there that are somewhat foolproof. My online backups go to Dropbox and Backblaze. I have had 100% recovery success with that solution. Granted, it was always test scenarios, but it worked.
Petréa Mitchell • April 30, 2013 2:51 PM
It’s real. Or the Rutgers police say it is, at any rate…
Jeff Morrissey • April 30, 2013 3:03 PM
In my experience over the last 20 years, 80% of the time people assume they have a valid backup from a particular point in time, do not. They may have a backup, but not a recent one… This hasn’t changed much in 20 years.
Paul Coddington • April 30, 2013 3:14 PM
Implementing backup and testing it with a bare-metal restore is the first thing I do when purchasing a new computer and setting it up. It amazes me when I see “professionals” claim they reinstall Windows every month (which means they do not know how to look after their machine, manage their time, and do not understand how backup works).
I’ve also seen cases where “professionals” (outsourced for high fees) lose massive amounts of other peoples data (government departments) where their backup policy was never tested until the RAID arrays died (through not having their batteries checked and replaced). And others who insist that restore will be “right on the day” in response to hard evidence of backup failure (logs, which they ignore).
djm • April 30, 2013 3:14 PM
The two laws of computing:
Attributed to Anonymous, by Fred Davidson, in “Principles of Statistical Data Handling”.
MrMishMash • April 30, 2013 3:17 PM
Since he wasn’t smart enough for backups, lets assume that he reuses passwords too…
Nick P • April 30, 2013 3:32 PM
I think a recent news story is relevant to this. A German media outlet reported their Dept of Education had a Conficker infection on a large number of machines. They guessed cleanup would cost around $130,000 and it would be cheaper to buy new machines. So, they just threw the old machines away and bought new ones for $170,000! (laughter)
I think one of the obvious cost reducers in these situations is a good backup strategy. It would have turned their decision from “how do we clean up the virus!?” to “which backup to restore to?” Data deduplication might help keep costs and time down if they have many nearly identical machines. There’s also quite a few [affordable] solutions for maintaining a business backup program.
In personal computing, I can’t say how often having a set of backups has helped drastically reduced the cost of recovering from both software and malware issues. I’ll add that some Linux desktop distro’s can be remastered into an install disk/USBdrive that has all the needed software. If the system and DATA are on separate partitions, it can be faster to reinstall fresh and just restore settings saved onto DATA partition.
A clever system setup can also allow you to use the web to accomplish useful work while the reinstallation happens on the disk. I truly wish my Windows systems could do that. 😉
Nick P • April 30, 2013 3:45 PM
Bruce’s rule could also be stated generally: nobody wants to do due diligence to protect themselves, but everyone wants to be safe.
Michael. • April 30, 2013 4:06 PM
Not only backup your thesis, but don’t backup onto floppy discs. Because they will fail when you most need them.
Wait, sorry, wrong century.
Err. So, don’t rely on one method of backup? Don’t trust the cloud alone, but also have an external HD, I think that’s the advice for a new century (or decade, whatever).
(Though to be honest, while I did use 3 1/2 inch floppy discs quite a lot, I don’t recall ever losing work because of one of them failing.)
Victor Engmark • April 30, 2013 4:09 PM
Seems that using lightweight version control could be a useful way to avoid non-working backups. Even just being able to see how often a file has been changed and by whom could give useful information to future users, in addition to essentially testing the backups continually.
Paul Renault • April 30, 2013 4:25 PM
Heh, I’ve been backing up for about two decades…
I just came across one of the tapes I used to use: a Travan 3.2GB tape. Tons of storage there, eh.
Tim S • April 30, 2013 5:07 PM
Say, from his password do you think he might have been born on 7/13/1985 and have initials zd? Second the motion from MrMishMash that this is probably his Facebook password, online banking password, etc.
Clive Robinson • April 30, 2013 5:08 PM
A question for all of you,
How do you know what’s on your backup is realy there and that you can use it when you need it?
It’s a question few ask and when I have asked other people about it I get answers that have more and larger holes than a second hand pair of string underpants.
It has a realy quit nasty little problem that an insider can realy ruin your whole organisation with.
Think about the actual process of how data is collected off of each piece of storage media and put onto a backup media, then how it gets back from the backup media onto the storage media.
Look for single points of failure/attack. That is programs or hardware that are used in both directions. Conceviably that point can lie to you and you cannot detect it…
I developed a proof of concept attack [1] many many years ago to prove the point (people don’t listen when you tell them you have to prove it to get them to believe it). What it did was at a key point encrypt the data going to the backup tape and decrypt it on the way back. As long as the tape was used in that system and the crypto key was not changed all looked well. You would only detect that the tape was encrypted if you took it to another setup which did not have either the decrypt function or the correct crypto key.
Thus after a while (ie your full backup cycle) your entire backup system becomes a hostage to the crypto key. If it’s lost or deleted then your backup tape is probably only usefull as land fill to those running the fire sale on your now defunct company…
So what do you do about stopping that insecure system admin from installing such an encryption system in your backup process and then deleting the key if you fire him?
Further for those that do have a guenuine encrypted backup policy what do you do about KeyMat handeling?
Remember yellow sticky notes have a habit of firstly droping off the tape casset with time and also some pens ink has a habit of fading quite quickly with time or other storage issues…
Doing backups properly is a very hard task that few if any actually check properly (remember that no two tape drives are the same etc etc). Worse things change in non obvious ways that can render an existing system usless [2].
Worse “secure backups” have all sorts of other issues on top of the plain backup problems that can easily make the previous world of pain look like a mear itch in comparison.
[1] It was a very minor change to a single script in a secure backup project that was developed for a customer. But it served as a valuable demonstration tool to other customers.
[2] Many years ago I used to work in the Petro Chem industry and some sources of the basic commodity can be in some fairly inhospitable places especialy when winter comes. One place I had dealings with collected a lot of data that was put on tape and sent back to head office via a courier etc. Well for various reasons the on site sysadmin thought it would be a good idea to make duplicates and take them home untill he had confirmation from head office that the tapes had been received and checked out OK. Well one day the inevitable happened head office did not receive a tape and the local storage decided to crash the heads. No worries he had prepared for this and brought the spare tapes back from home to discover they were faulty. When he had setup his system it had been summer when it went wrong it was winter and he found out that the reason his copy tapes were junk was the electric heater in the car seats of his car had acted as a tape eraser… As you can guess his boss was not impressed not because he had setup the system unofficialy because the boss would not pay for a proper system, but because the sysadmin had not checked his system properly even though it was unofficial and at his own expense. Which only goes to show some people know how to make the buck stop else where with extream prejudice.
Psandy • April 30, 2013 5:52 PM
As always.
Backup is free, its the restore capability that costs money
thabob • April 30, 2013 8:38 PM
That guy (or girl) does not deserve a PhD.
Figureitout • April 30, 2013 9:32 PM
@Bruce
–First off, that porn star at the top really looks like a nerd lol; she’s not even looking at the screen & I doubt she can type one-handed w/ her hand not even on keys lol. But seriously, you didn’t mention someone physically stole his laptop. Not even a flash drive? This isn’t even “APT” types conducting hacks that aren’t supposed to be possible. Coming from someone who’s had his “fun” (as a kid who was morally stupid and immature), (some b*tch tempted me recently, but that’s a separate story & never again) the amount of paranoia to remain “secure” is unacceptable; at least digitally.
So, if you really care about the data, make physical copies (and double check it b/c I hate most printers!), (recycle the paper). Someone will have to burn this ‘mother down to destroy or totally steal (original creator will still have a copy).
Steve Jones • May 1, 2013 1:36 AM
Yes backups are important, but probably more important is testing your restores.
One of my client sites thought they were doing everything right, backing up all their databases and servers every night. Sadly, one day they needed to restore something and they found that the most recent backup was corrupted. No problem they thought, move to the previous one. Of course, it was corrupted too, and all their other backups as well. The whole system was useless, as it exposed them to a single point of failure, which was a faulty tape drive that didn’t record data properly, but reported no errors.
Another client had a tape backup, but it was so useless that it didn’t do random access. So, to restore a file it had to rewind the tape and “play” through all the previous backups until it got to the one you wanted. Bizarrely, the tape drive was so slow that it took over 24 hours to find a file, by which time the latest backup was scheduled, so had to be suspended, putting everything at risk.
Of course, these anecdotes are to do with ancient tape backups, but the point about testing restores stands.
Martin P • May 1, 2013 2:59 AM
Steve.. Another ancient anecdote to support your point about testing – I once put together a backup regime on a VAX VMS system (yes, that ancient). I had a really clever (I thought) file naming convention that allowed – in theory – a rapid restore. The problem – only discovered when a restore was required in anger – was while file names up to 24 characters were allowed in the backup process (my file names were 18/19 chars long IRCC) – the restore process only allowed 16. All files irretrievable.
PJ • May 1, 2013 5:47 AM
Go into ANY university around dissertation season, final project submission time, and you will find notes like this pinned up. Lost USB drive, stolen laptop, mislaid SD card, you name it.
How these do not prompt people to make backups if they haven’t already just defies belief.
Jan Willem de Vries • May 1, 2013 5:53 AM
Years ago I read the following website: http://taobackup.com/index.html.
I think it is still valid.
wiredog • May 1, 2013 6:09 AM
One of the cool things about OSX is Time Machine. I’ve had to do restores from Time Machine a couple of times, and it was amazingly easy. Plug in the drive, tell Time Machine to use that drive, restore. When I bought the new Mac Mini, replacing the 24″ iMac, I just plugged in the drive, told it to setup from the Time Machine backup, and let it run.
I also have two USB external drives that everything in /User is copied to, and every month I go to the bank and swap out the one in the safe deposit box for the current one. Never needed that one, but it’s a good feeling to have it.
Brett • May 1, 2013 6:47 AM
All this brings something back to my memory. I remember starting to work at a company where I was the only IT person for our division. Not huge but we have five plant sites in the US.
There had been an IT guy before me that had setup the server and backups at each location. He even setup PC Anywhere and had a dial-up modem (yes this was 1998) so the he could connect to each server and check the tape logs.
Well I start, I check tape logs and see that all backups have been running and someone at each site replaces tapes on a dialy basis – all is good. I go to one our plant sites and after blowing 10 pounts of dust and fuzz out of the server is put in a tape and check what is on it. What do you know, not a single bit had been written to that tape, nor had a bit been written to the other 10 or so tapes there. But the log file showed everything was working great. I changed out the software.
So, I made trips to the other 4 plants. Guess what, same thing. In a year and a half of operation not a single bit had been written to the tapes. The guy before me installed it, ran it and only looked at the logs.
By boss just about had a cow when I told him. I think he realized how lucky they were to not need a restore in that year and half.
Long Horse • May 1, 2013 6:58 AM
@Clive
I developed a proof of concept attack ..
I did a similar design but never built it. Restores should be tested on separate equipment by other people.
After fire/flood/theft etc destroy all your hardware you will need to start again in any case. And using other people can be presented as making sure the documentation leaves no assumptions unstated.
Clive Robinson • May 1, 2013 8:19 AM
@ Jan Willem de Vries,
With regards the Tao it appears to have outlived it’s final head /purpose of pushing the Veracity data integrity backup solution 😉
I must admit I did use Veracity many moons ago (because I had to) but for the life of me I cannot remember what happend to it (I vaugely remember that the company got sold or some such).
And that raises an issue that people forget about which is what happens to your valuable data backups when the technology / software is nolonger supported…
Afterall backup software is just an application like any other which comes and goes in the blink of an eye with updates every few months. Likwise hardware for instance who remembers 100MByte Zip drives? What about Bernoli tape drives or 12inch optical WORM drives? As for mag tapes they realy don’t age well the state of the magnetic poles migrates through the carrier medium to produce echos and other degredation issues.
It has been pointed out that due to this you should copy your backups from one technology to another every year or two. But few doe this and considerably less realise that just copying is insufficient…
This is because there is another issue unless your data is stored in a truely open / known format it can be virtualy worthless in quite a short time. This is due to proprietry data file formats and rapid update cycles on them in major application software suites (which many have suggested is a way to force you to upgrade at great expense).
Oh and don’t think keeping old copies of the software is going to work for you… Because in most cases word proc and other WYSIWUG applications there’s the problem of hardware drivers for the likes of printers and even the OS etc.
I’ve always had a policy of storing data backups in the available format closest to plain text when you examine the actual data file. In the case of Word Proc docs on MessDross / Windoze boxes it would be the likes of Rich Text Format (I also keep Postscript printer files as well). Which has the advantage of enabling you to rip out the “raw text” using fairly simple software tools you can build with scripts on most *nix boxes. Thus even if parts of the file become corupted you still get the bulk of it back, which if you also have a paper copy alows you to rebuild it with minimal effort. Likewise saving spread sheets and databse files in Commer Seperated Value (CSV) format.
These psudo text files have other advantages, one of which is often you can check them in and out of your favourite Code Managment Repository which when making major documents like manuals and books can be a god send.
The point is thinking about backups is a major issue where longevity of the data is to be considered. Because you need to capture not just the raw bits and bytes, but the meta-data that turns them back to data, then the meta-meta-data that makes the data meaningfull, then the meta-meta-meta-data that restores the subjective meaning such as formating and layout positioning etc (hence the keeping of postscript printer files as well).
And to be honest I’ve found what is best at this sort of thing is what is in effect handbuilt solutions using standard *nix command line tools / filters and shell scripts. I can still recover (and have done) WordStar and Word Perfect pre Win 3 files and even engineering CAD files from the 1980’s. As such the systems I built have outlasted most (if not all) proprietry solutions I’ve ever used.
As was once pointed out to me many moons ago,
“There’s one heck of a lot more to being a good system administrator than being a Windows point and click pleb”.
TRX • May 1, 2013 8:24 AM
A former employer had both Unix and Windows servers on their network. I was the Unix admin. There were several Windows admins; they used a famous-brand backup package to back up to tape. I used tar and cpio. To my certain knowledge, they were never able to successfully recover any data from their backup tapes, though they continually bragged about how fast their backups were.
One day I found a box on my desk containing the Unix version of the famous software. The upper management had decided to “standardize backup practice” and have all the systems running the same software. I sent a memo saying I could save them some money by simply discontinuing backups…
I used daily tape sets, off-site storage, and “retired” tapes after a year. Once a month I restored a tape set.
It wasn’t a Windows/Unix problem, as much as the Windows machines were being run by “tech people” who didn’t understand the difference between knowing a lot about computers and knowing about system administration. And they were so blinded by marketing that even repeated failures of the product they’d picked didn’t constitute a “learning experience.”
Clive Robinson • May 1, 2013 8:31 AM
@ Long Horse,
Yup the documentation is another area where it can go horribly wrong.
One thing to watch out for is in Multinational organisations the people on site may not be as fluent in your language as you would like especially when it comes to language short cuts such as “pop it into the drive”, “fire it up” even “scroll across” can cause problems.
anonymous • May 1, 2013 8:56 AM
Sometimes people don’t want the university to have a backup to use the data themselves to start a new company after their diploma. Other than that I don’t understand why the university does not provide enough space for student to copy their files related to their research.
“A German media outlet reported their Dept of Education had a Conficker infection on a large number of machines. They guessed cleanup would cost around $130,000 and it would be cheaper to buy new machines. So, they just threw the old machines away and bought new ones for $170,000! (laughter) ”
They were going to replace them later. They only decide to do it sooner since they had a lot of work to do on the old one.
Bill • May 1, 2013 9:29 AM
I was visiting a school district technology office once years ago. One of the teachers called, completely irate because a tech re-imaged his district owned computer, and he lost his thesis for a Master’s degree.
Backup? Yeah, right.
jjjdavidson • May 1, 2013 9:48 AM
When our company first upgraded from individual PCs to an actual network server, we paid supposed professionals to set the server up for us. After they’d finished, I started checking everything myself (teaching myself Windows SBS on the fly).
It took me a couple of days to notice in the logs that the daily server backup (to an external USB 1.x port) ran in just four minutes.
These professionals (one of the largest commercial computer companies in town) had somehow set the backup job to back up C:\Documents and Settings\Administrator* instead of C:*.
At least my IE cache was safe.
h4x • May 1, 2013 1:29 PM
Govt of Canada kept all their records of student loans and other backups of financial and personal info on removable HDs. No surprise somebody lost or stole them and now there is a massive class action lawsuit because none of that info was encrypted.
MingoV • May 1, 2013 5:40 PM
A lawyer who used a Mac Plus (back in the 1980s) kept all of her applications and legal files on a single floppy with no backup, despite repeated nagging from others. Naturally, there came a time when the floppy didn’t work. I couldn’t repair the floppy, but I recovered the text of most of her Word documents. I gained experience at disk recovery, and, after hours of reformatting documents, she learned a tough lesson about backups.
Gweihir • May 1, 2013 5:57 PM
One more story from when I did my PhD: We were using IBM DeathStar disks in most computers (not mine, I bought my own disks and hardware) and one young lady did not have a back-up when hers broke. The choice was to pay about $8000 for recovery, or redo several months of work. She went with redoing it as her professor decided that backup was a basic and obvious requirement and she would have to pay for the recovery herself.
The really stupid thing here is that we had all the facilities, including network file servers and a very nice IBM tape library with a really simple interface. For backup, you just needed to copy the files to a certain directory. Despite this, she kept her work on her local disk for some reason.
Dirk Praet • May 1, 2013 6:03 PM
The sad truth is that most users as well as SMB’s are absolutely clueless when it comes to backups and restores, let alone a decent disaster recovery strategy. So are many consultants and IT consulting companies that are charging their customers serious amounts of money for hardware, software and services that in many cases are just utterly useless.
Over the years, I have lost count of the many companies I know that went out of business or got into very serious trouble over failing to recover/restore their data although they were 100% sure they had done everything right. Both customers and consultants focus way too much on hardware and software instead of on procedures, and that’s basically where it always goes wrong. The only correct approach that has ever worked for myself (and my customers) is the following:
1) Make a full inventory of all your data, categorise and classify them.
2) Conduct a company-wide business impact analysis to determine recovery time and recovery point objectives for both data and services.
3) In parallel to the above, conduct a decent risk analysis of all factors – internal and external – and the likelihood thereof that may in some way or another put your data and services at risk.
4) Choose your backup/recovery technology in function of 2/3. Consider adding fail-over and redundancy capabilities budget permitting.
5) Write down your backup plan. Talk it through with company management and department heads.
6) Implement your plan with the chosen technology.
7) Fully DOCUMENT all backup and recovery procedures and scenarios.
8) Perform a complete dry run disaster recovery of all mission-critical data/services either on-site or somewhere else. You have NO backup until you have successfully completed a restore.
9) Have a second dry run performed either by a different company or by other staff than those who have technically implemented and documented your disaster recovery strategy. The persons charged therewith need to be able to execute any scenario given using only the system/network diagrams and backup/recovery documentation.
10) Do targeted test restores/recoveries at least every six months.
11) Continuously evaluate and adjust procedures where needed.
As for private persons, I have always found the average disposition to backups pretty similar as that of smokers to lung cancer. Everybody knows that not having backups sooner or later will bite them in the ass, but you just burry your head in the sand hoping that it will happen to someone else and not you. Time Machine really couldn’t be any easier for Mac afficionados whereas Windows and Linux equally offer plenty of utilities – FOSS and commercial – to reliably back up data or entire systems. With external hard drives and storage capacity being as cheap as they are today, there really is no reason not to do it. And if you can’t be asked to figure it out for yourself, simply ask a relative or the friendly neighbourhood geek in the local pub or gym to do it for you. It will probably not cost you more than a home cooked meal and/or a bottle of Jack Daniel’s.
WL • May 2, 2013 2:16 AM
A (now) PhD I know kept a copy on a laptop (where most of the work took place), a RAID at the lab, a cloud storage provider, and a pair of USB sticks (rotated so one was always at home). She really didn’t want to lose that data…
Glader • May 2, 2013 6:07 AM
I’ve seen a couple of these on campus. If it’s not the entire laptop it’s a USB stick…
Doug D • May 2, 2013 10:30 AM
“If you are not testing restores, then you are not performing backups.”
Gweihir • May 3, 2013 5:44 AM
@Doug D: Depends. If it is a document or code backup, a full compare is good enough. If it is in version control with checksums, checking those on the backup and verifying the checksums against the original is also good enough. What is not good enough is to never read the backup or verify (in some way) that what you expect to be in there is actually in there and reads correctly.
htom • May 12, 2013 12:33 PM
“It’s not backed up until it has been restored from that backup!”
Watch out for the sneaky computer that restores garbage, correctly reporting that what was restored matches the backup, leading you to infer that the garbage matches the original.
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
Steven • April 30, 2013 1:35 PM
I’ve known many IT shops where the backups worked better than the restores…