Schneier on Security
A blog covering security and security technology.
« Lie Detector Charlatans |
| MI6 and a Lost Memory Stick »
May 7, 2009
Virginia Data Ransom
This is bad:
On Thursday, April 30, the secure site for the Virginia Prescription Monitoring Program (PMP) was replaced with a $US10M ransom demand:
"I have your shit! In *my* possession, right now, are 8,257,378 patient records and a total of 35,548,087 prescriptions. Also, I made an encrypted backup and deleted the original. Unfortunately for Virginia, their backups seem to have gone missing, too. Uhoh :(For $10 million, I will gladly send along the password."
Hackers last week broke into a Virginia state Web site used by pharmacists to track prescription drug abuse. They deleted records on more than 8 million patients and replaced the site's homepage with a ransom note demanding $10 million for the return of the records, according to a posting on Wikileaks.org, an online clearinghouse for leaked documents.
Whitley Ryals said the state discovered the intrusion on April 30, after which time it shut down Web site site access to dozens of pages serving the Department of Health Professions. The state also has temporarily discontinued e-mail to and from the department pending the outcome of a security audit, Whitley Ryals said.
More. This doesn't seem like a professional extortion/ransom demand, but still....
EDITED TO ADD (5/13): There are backups, and here's a Q&A with details on exactly what they were storing.
Posted on May 7, 2009 at 7:10 AM
• 63 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
How does this idiot think he's going to get away with this? How did he manage to destroy their backups? Why werent the patient records encrypted (Or were they?) Lots of questions at this point.
Holding data ransom works even if the records were encrypted (which, given the usual record of such things, seems unlikely).
Ballsy. I don't think I've ever seen anything quite so brash. Maybe when the feds inevitably catch the people responsible they'll nuke 'em.
Just how could a hacker remotely erase backups that should be located on a tape, in a fireproof safe, in a separate building?
Well if Somali pirates are still in business - this surely is at par.
I wonder if the state of Virginia will fire anyone or take their pension benefits away for sleeping on the wheel.
It is really a painful. How can do anyone like this, with patient?
And which types of security is providing by Virginia states?
Now, what will be happen..?
How can resolve it?
Lots of question at this point.
Concerning the destruction of backups:
I i had been that hacker (which i'm not!) i would have corrupted the backup-routines and waited for a few months.
No-one ever tests the backups.
"Why werent the patient records encrypted (Or were they?)"
It is not common practice to encrypt data while it is on the server, especially large chunks that would be accessed by hundreds if not thousands of users simultaneously.
"No-one ever tests the backups."
While I understand (hope) that was a bit sarcastic and rather over generalized... most back up systems that are in use have automated alert systems (sms, email) built in that one would hope even the mediumest of companies would employ.
There is something about the amount $10M that keeps popping up in stuff like this and it almost seems like twenty-somethings are fixated on that amount.
Now we might be reaching for the tin foil hat here, but this could be another "ploy" to get the general public closer to a thought about the government agency to patrol the interwebs. Perhaps a mock ransom ploy to support the need for say... an NSA watch dog?
There do seem to be a number of unusual weaknesses in the system.
While it's certain that something is going on here (there's no question that a ransom demand was made; but whether the data is both unrecoverable and in the sole possession of the ransomer is unknown by the rest of us), it seems unlikely to me that there's much more than a defaced webpage here.
Mostly, because I'm sure that the data is worth much more than $10M on the open market. He's barely asking for $1 per record!
Unless he plans to ransom them AND sell it himself; after all, whether he sells it or not, they still need it back. But if his story is the truth, he probably has no intention of returning it for any amount of money (makes it far too easy to get caught); the ransom demand would simply have been a way to advertize his newly-acquired data to the appropriate interested black-marketeers.
@How did he manage to destroy their backups?
If the backups were automated, and were singular backups (ie; not grandfathered/distributed/patterned etc.), then simply corrupting all of the data 1ms before the scheduled backup time would cause the backup to be overwritten with rubbish.
Also, that *the ransomer* made a backup does not imply that they ever did. I seem to remember the occasional argument that if you encrypt someone elses data but don't actually destroy it, you're only technically guilty of the lesser charges of Computer Trespass and Denial of Service. I'm not sure if this is the case; the idea could be purely theoretical as I don't know if it's ever been tried (the charges are dropped for lack of evidence more often than not). It hardly makes much of a difference if you add crimes like Extortion on top though :-)
Most likely, it means that he only has to provide them with a password to recover the data, rather than providing an unencrypted copy, which would likely require a physical interaction.
An insider would find it easier to destroy backups than somebody hacking in from outside, so that would be my guess. An outsider would need to know something about the operation to make sure that all backups were bad or destroyed, suggesting inside contacts.
So, I rather expect the extorter to be caught, and sent to prison for a long time.
It would be nice if everybody would handle backups intelligently (including the motto "it isn't a backup if you haven't tested a restore"), but that doesn't happen in real life.
If you were a Fed who wanted to freak people out about the dangers of hackers, what would you do? And how would you get away with it? Well, let's start here: you're a Fed.
I wonder if ol' Bunion Butt Limbaugh is in that database... or any right wing politicians who can freak out appropriately to the media about this newest piece of Internet sky that has fallen.
And on Obama's watch, too. Clearly Obama needs to feed his Police State machine better. Tsk tsk.
This is the sort of thing he could get away with if he's really careful to leave no trace.
Unfortunately for him, receiving $10 million is incompatible with leaving no trace.
Unless he drops the whole thing and walks away now, I think he's inevitably headed for a long prison sentence. Whether VA gets their records back is still an open question, though.
"If the backups were automated, and were singular backups (ie; not grandfathered/distributed/patterned etc.), then simply corrupting all of the data 1ms before the scheduled backup time would cause the backup to be overwritten with rubbish"
True, but if you run daily and monthly cycles. You can run the tape from last week or last month...
The only way around this is to corrupt the date over a year ago, and keep corrupting new data, but then nothing would ever be visible and it would be blatently obvious
As a person who objected to their medical records being put in electronic form (in writting to UK Gov) for exactly this sort of reason. And had my wishes over ridden and then lost by St Georges Hospital in London and again disclosed by insecure systems of "Chose and Book" combined with the bad practice of a bunch of profit minded UK doctors who set up KCAS.
Further the UK NHS Spine which is supposed to be the worlds largest ICT project (that BT Bruce's employer and others have significant involment with) is known to be a significant distributer of malware to hospital systems. This is because they have to trust it and it is so poorly designed security wise (NHS staff get Smart Cards but their password is their Date of Birth) and no sensible audit trail is built in and numeours below security101 mistakes you must question senior managments ability and sanity in such involvment.
Therefor I already feel the pain of the people in Virginia.
For those who are going on about backups,
1) who checks backups?
2) who checks them on another machine?
3) who checks them on a known secure and clean machine?
To make them disappear is the same trick. Get into the systems replace the backup prog or insert a modual in the device stream that ciphers the data. As long as you get all the mchines in use then all looks normal to step two until the person deletes the key or modual then good by backups.
This not an unknown attack and is usually discussed indirectly from the oposit direction when formulating external backup policy (ie to encrypr or not and keymat issues).
So not exactly difficult to do and very easy to pull off on any collection of systems where malware can get.
When will managers buracrates and politicians SMELL THE **** COFFEE...
Sorry the pervious post comes across as very angry and a swipe at you.
The first is true the second is not.
With regard the first I have litterly just come out of hospital having been very very close to being dead, the root cause of my illness is almost directly attributable to these brain dead medical ICT policies in the UK. And the stress caused by ID theft arising from them.
Oh and those responsable that sit around bleating about it's never going to happen and acusing those they see as neighsayers as being "chicken littles".
Well the sky has fallen in on some of us and we have been provably hurt well beyond any measure required for criminal prosecution, or for malfeasence in public office.
It's just that we still don't have a generalised "sue for gain or revenge" ethos in the UK and some of us want not only to stay that way but more importantly we want the system put right without crippeling that possability by taking all of the available money away from it in legal expenses and damages.
With regards the second issue,
Personaly given the back ground I think you show a great deal of independance and integraty even mentioning the subject.
We have an expression in the UK about "having a dog and barking yourself" and I don't envy you the possition your employer has effectivly put you in.
You would think that if an organisation get the expertise on board you would atleast expect them to use of it on the highest risk project they have involvment with.
But then again I'm an left-handed engineer, I live inside my head not that of others and thankfully have no (current) need to subscribe to the "live inside others heads" that society holds so high.
Glad you're around. Aside from the general desirability of minimizing premature mortality, your comments are always informed and thought-provoking.
The hacker might just be out on a limb with his assertion that he "destroyed" the backups. Maybe he just assumed that the backups were as badly managed as anywhere else.
Or he was more sophisticated in that he discovered how the backups were being made and tampered with them. E.g., if he found the script and learned that there was a weekly full backup with daily diffs, he could have changed the script to back up zeroes and waited for a week.
All speculation, of course.
On what basis do you say that "this doesn't seem like a professional extortion"? Do we have enough information either way?
Certainly if the hacker doesn't have any prior experience with extortion, he will have a difficult time getting away with his cash. A demand of this size presumably makes a lot of resources available to law enforcement. I don't think that you can just pick up a laundry bag with $10M at a dead drop and freely walk away with it. And he doesn't even have any hostages that he can hold until he and his money is safe.
@FP: the tapes could have all been "safely stored" in a backup bot. The attacker just had to do a
while backuptool --load-next-tape ; do backuptool --format --force ; done
I've had to fight a client over this already, he only agreed to do it (not sure if he actually does) when I asked him to sign a paper stating that I had informed him that leaving tape in a robot could lead to massive data loss.
I think the $10 million amount is what most people have in their heads as "F--- --U" money. As in, "If I had $X I would never need to work again, and could tell the boss 'FU'"
As a survivor of the attacks on the WTC on 9/11/01, I felt the pain re-generated by the USAF's thoughtless "photo op" flyover of lower Manhattan several days ago. However, just as with the original attacks, I don't let that pain blind me to scrutiny of ALL the possible causes.
Insider threats are real, and I am of the opinion that there are similarities between the flyover incident and this one. Obama needs to clean house. He has insiders who want to make trouble for him. The person who pulled this off is not some 20-something tard living in his momma's basement. The only way they'd get away with this would be if they were allowed to. I smell a Fed and yet another entirely contrived crisis designed to further justify a police state.
Clive, I hope you recover from your illness soon.
"Also, I made an encrypted backup ..."
Good thing he's securing his data. :-}
As I know a lot of the people working directly for, or as contractors to, the Commonwealth, ('though not in this particular program) it's the same story as elsewhere: Understaffed groups asked to shove in a system in an unreasonable amount of time. I think those of you criticizing them either don't spend any time in the trenches or are among the very fortunate few who are provided with plenty of resources to secure your systems.
In case you missed it, every state is working on or has a database like this. It's used to correlate data on narcotic prescriptions to make sure us nasty citizens aren't double-dipping and getting extra meds from different doctors. Right or wrong, it's the first step in big brother for medicine and will only get worse with online, government controlled electronic healthcare records.
Sounds like a job for the SEAL team snipers...
Suppose it would cost a mere $50 per patient to recreate the missing data. The sticker shock is bad: $413 million. They might see it as worth the risk to pay the ransom in hopes of getting their data released. They could get the cash together and keep the FBI in the dark. The extortionist no doubt has other channels to communicate with them, so if his inside man lets him know they're willing to pay, he can set up the payoff.
The company is likely to care more about keeping their jobs than catching crooks.
Does anyone else hope that the hacker gets away with the money for whatever reason?
@derf; Agreed, the measures the government put in place to keep us "safe" do far more harm, although in different ways (privacy), in the long run than if they just let people live free in this so called "free" country.
This is the wrong way to share confidential and private information. It's a bad business plan.
Hope things get better, Clive.
".. he is only asking $1/record .. "
Dude you must be working for an insurance company or a pharmaceutical.
Frankly - what's the big deal? This data is not heavenly -- it's just some patient records of what medicines they are talking etc. etc. State can ask them to resubmit the information and it will be ok .. it's not end of the world.
The trouble is patients will sue state for "emotional suffering" .. state employee will have to hit keys on keyboard again .. too much work - lets just pay the bums.
Seriously, why would you want to expose confidential data to the world? Or even state public data. Why would people outside of the taxpayer's jurisdiction have a right to view that data? The world wide net isn't the place for it. Why would the air force even want a server in China to have the ability to knock on their server's doors?
Sorry, maybe I'll just call myself 'fed-up'.
If he has any sophistication he's setup an offshore accounts. Someplace that isn't extradition friendly with the United States works.
Sounds like a childish prank to me. Some kid managed to get in do the deed and ask for $10m without really thinking about the consequences. I mean the ransom note is a bit "OMGWTF d00d" isn't it?
Of course, I suspect the authorities wouldn't see it in such a favorable light.
I think the language used is more Coen brothers' "Burn After Reading"-esque than anything.
Dr. Evil: I demand the sum... OF 1 MILLION DOLLARS. ...
I'm a VA resident, and this does not surprise me at all, given the quality of other VA computer systems administration. Try our DMV if you have any questions.
This is why I object strenuously to government statements that we must give them data, from Census to income, and it's fine because they will keep it secret (from whom, exactly?).
Sure...they can't keep it secret from other agencies, or the world at large, but I'm supposed to buy this? If a plain old citizen made this statement, I could refuse for any reason I want.
When the government makes it, what choice do I have, given the the government tends to make it very hard to sue them even for obvious malfeasance?
Lucky it was only data on dopers this time, who is next? I suppose these people could be targeted for theft, since it is now known where to go steal the drugs (common around here).
The government isn't the only one a fault here. Not that long ago, a newspaper "innocently" published a list of all firearms concealed carry permit holders, complete with addresses and so on. (they have a strong anti-gun agenda) This WAS used to target many who had gotten permits to protect themselves against stalkers when restraining orders didn't work.
There's your government guarantee.
"the ransom demand would simply have been a way to advertize his newly-acquired data to the appropriate interested black-marketeers."
Agree. And any other exploits in progress that aren't yet known.
You suppose that those records are going to be just a patient ID number and their prescriptions? Not a chance.
They'll have name, address, phone, etc.
If I wanted to push alternative medicines, cosmetic therapy, viagra, or other similar products, that list would probably be worth about $10 per record if it had all of the above information.
By the way; I just checked the official population for Virginia; it's less than 8M. Something is fishy here…
Yeah, this demand is absolutely a Burn After Reading reference. Being a bad guy doesn't mean you can't watch films and enjoy silly pop-culture references from time to time.
Could've been worse, could've been a /b/ meme.
Windows Server 2003 running Microsoft-IIS/6.0? Yeah, I can believe that'd get hacked. In the private sector, I'd fire the IT staff for making choices like that. In the public sector, people should be getting jailed over things like this.
Its a semi-hoax. the guy hacked a website and put up a fake ransom note. i doubt he has anything at all, but even if he did there's going to be backups at remote locations. The language was clearly meant to be a joke, and anyone who actually wanted a ransom would keep it secret. Publicizing it just makes it less likely you'll get paid and more likely you'll get caught. The best reason to pay this kind of ransom is to keep the embarrasing intrusion and security failure secret.
Spot on, and a great movie besides. I can't begin to explain how much I identify with the Malkovich character sometimes (not a fed, barely employed, just, *sigh*... when you spend enough time with arcane information, idiots can make life very complicated)
While I understand (hope) that was a bit sarcastic and rather over generalized... most back up systems that are in use have automated alert systems (sms, email) built in that one would hope even the mediumest of companies would employ."
Posted by: Anonymous at May 7, 2009 8:31 AM
It would be really easy to manipulate the backup-routines to randomize the names and IDs of the data. Most companies only check taht the backup-process did finish correctly and produces a reasonably sized file. Even if a human would take a look at the backup, the data in it would seem ok.
It's probably true that the hacker just invented those numbers. There are not 8.2 million people in Virginia, and even if you count people who have died and people (for example) from DC who go to Virginia doctors, it's not at all likely that you'd have 8.2 million patients in the database. Especially since this database is only supposed to contain records for 'certain types' of drugs.
The ratio of 4 'certain type' prescriptions per patient is not very plausible either, IMO.
I worked for the Virginia IT agency for a few months before leaving for a much better job. I consider this to be an entirely believable attack considering some of what I saw oing on during my time with the State.
The IT infrastructure in Va. is being taken over by Northrup Grumman. They are converting everything to their "vision" of a unified network, and cutting corners security-wise and equipment-wise in order to make additional profit. While I was there I saw them continuall overcharging for equipment, and doing the least work possible in rder to adhere to the letter of their contract.
The worst part is that Northrup Grumman has managed to build an organization, staffed by its own employees, that has the authority to override state employees who won't sign off on bad infrastructure designs. I saw several cases where state employees would refuse to approve a network design because of security, and/or scalability problems, only to have the project go forward because a Northrup Grumman employee was allowed to sign off in the place of the state employee.
Since I left just about everyone that I worked with has left for a better job. While I was there, the contractors had real problems getting anyone to staff the security team, and based on a quick check of open infosec jobs around the Va. datacenter, it looks like they still haven't managed to fully staff the securit team there.
There's a FAR FAR bigger issue here. Virginia's 2008 population estimate is 7.9 Million. Where the heck did the 8.2 Million number come from? Is Virginia tracking the prescription patterns for every one of us in the state?
The 8.2 million number will probably be the result of record duplication with typos, name changes, etc. Considering the corners that have obviously been cut I wouldn't be surprised at the lack of input validation!
I think you have to factor in people coming into and leaving the state. I wonder if there is a different entry per prescription.
"Just how could a hacker remotely erase backups that should be located on a tape, in a fireproof safe, in a separate building?"
My thoughts exactly.
I'm not surprised that there are more prescriptions than people (I imagine half of people are on one, and many people are on many).
But as I've been saying I'm more concerned that he could change some prescriptions. Changing someone's medication by 50% probably won't be caught by a pharmacist but could cause deaths.
"Just how could a hacker remotely erase backups that should be located on a tape, in a fireproof safe, in a separate building?"
You are looking at it incorectly it is not,
"Remotely erase backups"
"Make backups un usable"
Although the end result is the same on is easy the other is hard.
Making them unusable is very very easy and as I said above relativly easy to do undetectably in mot cases due to the way you test a backup (if you even test it at all).
To make any and all the data on the backup tapes made after a certain point in time does not require that you change them in any way.
Such is the magic of secure encryption, you just "throw away the key" and they are "effectivly locked up for ever" (which might be your fate if you do the next bit 8)
All the cracker needed to do was gain access to the data "stream" between the backup tape device and the the backup operaters interface software.
They insert a modual that encrypts all data after a certain point in time but decrypts it transparently upto some epoch date at which point it trows away the key.
To the operator this modual does not make it's self apparent as things still work the way they should on that machine. It is only after the epoch that the sky falls in on the operator.
I asked three questions in my earlier post,
1) who checks backups?
2) who checks them on another machine?
3) who checks them on a known secure and clean machine?
What the cracker has done is transparent so if you answer yes to 1 then you have lost to the cracker.
If the cracker has got their way into all your backup machines via malware then even if you answer yes to 2 then you have lost to the cracker.
If you have answered yes to 3 then you have beeten the cracker as the tape will show up to the operator as bad the first time a tape is encrypted.
I have worked in many different and sometimes strange and supposadly secure environments and I have yet to see an organisation that does three in a proper way...
So the answer to Juergen's question is not easy but not difficult let alone impossible.
I have actually talked about this in the past to people with everything they are and control thus their livelyhood riding on this very issue and when I explain it I get either "you don't know what you are talking about" or "it's not possible" or "it's never going to happen".
One such organisation managed the UK's NHS Net (National Health System Network) which was one of my reasons for writing my letter to the UK Gov.
Now the sky has fallen in on some people in Virginia, does it make me feel better knowing that It could be done?
No it just makes me sad that I could not convince another organisation of the reality of the attack 8(
"1) who checks backups?"
people who work to strict data retention standards such as the medical sector ...
First i think is more of a hoax that anything else.
But if its not, to those suggesting that its a good idea to pony up. What are you thinking! Is this person is somehow dependable, reliable or trust worthy? You pony up and you end up 10M down and *still* without your data. There ZERO reason to hand over key, if there is even a key to hand over.
Given budgetary issues, I've seen "Enterprise Backup" mechanisms that make little sense... but I've also set up (and managed) a backup structure using ADSM (predecessor to TSM) which, while not particularly doinkable as other systems, has its own triage issues... and setting up policies for database backups aren't really trivial.
One thing I am certain of, however... backups are "overhead" and therefore usually under-funded by bean-counters as an "unnecessary cost"... but, then, bean counters seldom can handle the concept of a "critical path" or "resiliency" since their brief is to maximize economic efficiency.
So, sure, there was a backup tape... but it was likely over-written enough times that the oxide is coming off (high capacity tape cartridges have a limited number of "passes", after all) since that was all the budget allowed.
Economically efficient systems-- where the bean counters forget that TANSTAAFL rules-- tend to be brittle if not as sharp-edged as a pile of dry sand.
I work with a system with a huge database which takes a BUNCH of Ultrium cartridges to hold and we've had cases where one of the tapes in the middle was bad... so the backup had to be repeated.
Taking backups imposes a price... unfortunately, when a bean-counter looks at the risks of NOT having an ability to restore the system, they may not consider the need to spend more than a pittance as worth the time and money.
Hey, guys, is it time to trot out the "Institute for Backup Trauma" again, for laughs???
Note to Moderator:
EDITED TO ADD (5/13): [there are backups] LINK is unavailable. Goes to site though.
I tried link a few minutes before this submission.
Thanks, PackagedBlue. I've updated the links.
The website data may be backed up, but it is irrelevant. The actual data is sent from pharmacies to the Virginia program. They will still have those files. Also, there are only 24 months worth of data which means the data ages out quickly. Finally, the extortionist should not be paid since there is no guarantee that the data has not been copied. Conclusion: It is a college student hoax.
This case is still being investigated. For three months I had an FBI agent playing games with my lawyers to answer questions about some case that I was not a target in. Finally, they served me with a subpoena and I have testify as a witness before the grand jury. I know little about this case except what I read on the internet. I think they are harassing me because I connected Aneesh Chopra to the hack on my blog:
since when does the government have a right to my prescription records? government spying on citizens, not my fathers country!
Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc.