Schneier on Security
A blog covering security and security technology.
« UK Police Using Military Drones |
| ITSecurity Interview »
May 25, 2007
Criminals Hijack Large Web Hosting Firm
IPOWER declined a phone interview for this story. But the company acknowledged in an e-mail that "over the past three months our servers were targeted. We take this situation very seriously and a diligent cleanup effort has been underway for many months already. We saw the StopBadware report on the day it came out and went to download the list to sweep it as quickly as possible. By looking at the list, it was evident that our cleanup efforts were already helping significantly. By the time we downloaded the list, there were already over a few thousand accounts less than what they claimed in their report."
IPOWER said the site hacks "came from a compromised server hosted by another company that was listed on the Stopbadware.org Web site. This impacted a higher percentage of accounts on each of these legacy third-party control panel systems."
The company claims to have more than 700,000 customers. If we assume for the moment the small segment of IPOWER servers Security Fix analyzed is fairly representative of a larger trend, IPOWER may well be home to nearly a quarter-million malicious Web sites.
And an interesting point:
An Internet service provider or Web host can take action within 48 hours if it receives a "takedown notice," under the Digital Millennium Copyright Act. The law protects network owners from copyright infringement liability, provided they take steps to promptly remove the infringing content. Yet ISPs and Web hosts often leave sites undisturbed for months that cooperate in stealing financial data and consumer identities.
There is no "notice and takedown" law specifically requiring ISPs and Web hosts to police their networks for sites that may serve malicious software.
Posted on May 25, 2007 at 7:13 AM
• 23 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
It looks like the old comment about,
"You get what you pay for"
Applies to online hosting and ISP sites.
The real question is how many people putting up websites with these organisations actually have the skill to secure / check / clean up their websites?
I suspect very few, however does that make the hosting company liable probably not in their view (others will certainly disagre).
However when the site operator is deficient in applying security and other patches then they most definatly are deficient and should clean up their act pronto (as IPOWER apparently now are).
There you have it. Copyright lobbyists have more pull than computer users, or simply people.
@Clive Robinson: I am not quite sure that law should be a paid-for good.
I manage several web sites for various organizations. I have *some* sites still hosted on IPower, but I started migrating away from them last year due to poor technical competence and not meeting their customer service claims.
A complicating factor was the IP for one of my sites is in various blacklists due to the behaviors of people sharing that IP. Despite numerous emails, IPower seems decidedly uninterested in actually solving the problem. I don't know if this is because they didn't know how or because they were actually enjoying the profit from *those* customers.
In any case, they no longer enjoy profit from me.
Unfortunately, this is NOT an isolated incident, and iPOWER is NOT alone.
While researching a malware that was being seeded by hacked websites with hidden iFrames I came across close to a dozen hosing providers that were in the same place as IPOWER. Some of them I could get the sites fixed, but some of them I could not.
In addition I found several (half a dozen) hosting providers that had hidden sites for the malware authors to use for various purposes.
In one case, there was over 2 gig of raw data from compromised machines sitting on a hosting provider server that the provider knew nothing about.
This issue is a pervasive issue throughout most hosting companies, with relatively few exceptions to the rule.
A numerological curiosity, which certainly has more to do with virtualization than with security: the nine virtual servers chosen for examination by SecurityFix comprise 8192 websites -- that is, 2^13.
I suppose this would be unremarkable, if they weren't partitioned so weirdly -- nine virtual hosts (why not eight?), and a highly inhomogeneous distribution of website numbers over hosts (the rightmost column in the figure, which adds up to 8192).
Also, irrespective of the IPOWER setup, how did SecurityFix pick the virtual hosts to analyze? They must have seen through the weird partition somehow, and gotten an exhaustive list comprising the 2^13 websites. It's not discussed in the article, possibly because it's only interesting to, well, OK, I'll shut up now.
I picked the virtual hosts by locating CPanel6 as the reverse DNS entry for the first site I found on StopBadware's report that was hosted at IPOWER. From there, I started working backwards pinging hosts, CPanel5, CPanel4...etc. until I had a nice list of responding Cpanel servers.
Hope that helps.
Larry O'Brien wrote about his root-kit issues with an IPOWER server 2 months ago. Also a nice commentary on their "security capabilities".
"@Clive Robinson: I am not quite sure that law should be a paid-for good."
Not quite sure how you came to that conclusion.
What I was intending to say was that from the report it appears the less you pay to these companies the less you get in return and that they are not the bargins they might appear to be in the adverts (much like many other things in life).
The obvious assumption to make is that the price reduction is due to the savings that have been made in the "human" resources in the SysAdmin areas. Be it in numbers caliber or training.
I guess I also did not make clear the distinction between the two security senarios I was talking about.
Most security people have come to the conclusion that the majority of people writing software cannot do it securely (I guess you could say this was evidenced based reasoning).
So if a hosting company provides PHP as part of the "paid for" package then they should be responsable for keeping that code current (which is what I would expect them to do as a minimum along with backups etc).
However if I wrote insecure PHP code and put it up on the hosting companies server it would be a bit much to expect them to be responsable for my broken code.
I would however expect the company on discovering I had put up insecure code to atleast take some action (such as notifing me / taking the virtual site I was using down etc).
oh, and I forgot.....I could not have done this project without the help of the excellent tools over at Domaintools.com (the paid version).
I just moved my site from iPower after four years because of a sudden bout of sheer incompetence on their part, specifically an unexplained "server migration" that happened with no notice or warning, but left my site inaccessible for two days. Given their rank amateurishness, I'm not at all surprised they're having this trouble as well.
What a shame... My web hosting guy knows his stuff :)
Is there some reason that one couldn't use a Digital Millennium Copyright Act takedown notice to have a site removed?
Would anyone bother to dispute a claim of authorship of an exploit?
If you want to know how far back this goes, google 'ipowerweb hacked iframe' and you'll see this isn't a new issue at all (it goes back to 2005 at least). Also, if past behavior is an indicator of future behavior, I wouldn't believe their claims that they are cleaning up, because IPOWER has constantly been hosting a large number of compromised sites; they've never been able to get their hosted sites cleaned up. Typically they claim it is the user's responsibility to maintain their own site, or they'll blame the user for using a weak password. I find it amazing that those of us outside their network can find thousands of infected sites that they host, yet apparently they can't run a search on their own servers to find the sites and fix them. Even if we believe their typical excuse that the compromises are due to weak passwords, you'd think they would notice a brute force password attack against their servers.
something which is not clearly stated is that, apart from infecting the webpages, the attackers launched a sort of DoS attack to the Web hosting company itself since StopBadware.org registered it to its lists.
"He said the company told him that it was his responsibility to maintain the security and integrity of his site."
It makes no sense, but if you give it second thoughts it might actually make. If I upload a webpage with a malicious script, or with illegal content, will the web hosting company be liable of or me?
Is there any relevant legislation in the U.S. or it differs according to the contract the two parties sign?
iPower was aware of the problem months ago. I don't know if I brought the problem to their attention; follow the URL in my sig for the story (posted in March).
@Carlo Graziani: the ninth host is for parity...
I was wondering if this is the same virus that spreads like a .js script. Recently (like a couple of months ago...) found about it. It turned to be loaded into the kernel (?!). Loading a grsec fixed that and no sign from that 'virus' anymore.
Worth mentioning is that the BSD boxes were not affected.
Sorry if that is a bit off-topic, but was just curious.
Getting customer service -- or a real human to respond -- from iPowerweb is worse than pulling teeth. If they weren't cheap, and my site needed more than I can easily handle, then I'd be gone in a flash. The vast majority of the e-mail to my admin account is from my own domain or autobots responding to spam from my domain. I've complained, to no avail.
This is another reason to avoid iPower.
I'm not saying that this is the case, but it is an interesting hypothetical to imagine:
If I were a "bad guy" operation that had significant interest in computer SPAM/phishing/zombie activity, it might make sense to set up a front company that offered low-cost web hosting. Keep a pretty good "cutout" between the front company and the real "operation", and ensure that the front company never really has the funds to put together a high-quality sysadmin/net-security team.
Deliberately plant weak-security sites to be hosted, and "sell" access-information to others. For the really important stuff, have your inside person provide direct access to the underlying O/S (so you can infect/alter/whatever any hosted site, without cracking passwords).
But, otherwise, the front company just
acts as a "normal" hosting service. If most of its employees have no idea what is "really" going on, it might take quite a while for anyone to prove otherwise...
I ran a samll hostign company for a year or so, and while I think a lot of the blam falls on Ipower (how do you not keep your servers patched up?) I can feel their pain. You can harden your servers as tigh as possible, then one users has abadly writen pice of PHP and blam the attackers have shell access, or a nice spam relay
This is so stupid, 99 percent of the people that got hacked was due to their weak lame password.
Make your passwords stronger and this wont happen.
Same thing happened to my sites. Trust me it was not from a weak password (a combo of 10 upper case & lower case letters, symbols and numbers). I am on a dedicated server and my sites are straight HTML. Of course my hosting co. gives me the line about a weak password....
Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc.