Most Spam Came from a Single Web Hosting Firm

Experts say the precipitous drop-off in spam comes from Internet providers unplugging McColo Corp., a hosting provider in Northern California that was the home base for machines responsible for coordinating the sending of roughly 75 percent of all spam each day.

Certainly this won’t last:

Bhandari said he expects the spam volume to recover to normal levels in about a week, as the spam operations that were previously hosted at McColo move to a new home.

“We’re seeing a slow recovery,” Bhandari. “We fully expect this to recover completely, and to go into the highest ever spam period during the upcoming holiday season.”

But with all the talk of massive botnets sending spam, it’s interesting that most of it still comes from hosting services. You’d think this would make the job of detecting spam a lot easier.

EDITED TO ADD (12/13): I should clarify that this is not the site where most of the spam was sent from, but the site where most of the spam sending bots were controlled from.

Tags: botnets, economics of security, Internet, spam

Posted on November 17, 2008 at 5:11 AM • 25 Comments

Comments

Muffin • November 17, 2008 5:43 AM

“But with all the talk of massive botnets sending spam, it’s interesting that most of it still comes from hosting services.”

The spam didn’t come from McColo – it was sent by a botnet which was controlled through machines hosted there.

As the article says: “a hosting provider […] that was the home base for machines responsible for coordinating the sending of roughly 75 percent of all spam each day” (my emphasis).

Rick DeNatale • November 17, 2008 6:11 AM

I don’t know if it’s a coincidence but…

Over the past few months, I’d noticed a big drop off in spam comments to my blog.

Then shortly after the McColo shutdown, the blog-spammers seem to have come out of the woodwork again.

There’s probably no connection, but perhaps the spammers are looking for an alternate channel.

Sandi Hardmeier • November 17, 2008 6:14 AM

re “But with all the talk of massive botnets sending spam, it’s interesting that most of it still comes from hosting services. You’d think this would make the job of detecting spam a lot easier.”

Nope. McColo simply controlled all of the spambots out there … when the spambots were left with no way to ‘phone home’ for new instructions they went silent.

Trichinosis USA • November 17, 2008 6:33 AM

“The world saw a similar — if short-lived — drop in spam volumes in September, following the demise of Intercage, a.k.a. “Atrivo,” another Northern California based ISP that security experts identified as a major source of badness online. In that case, it only took the spammers a few days to find a new home. It seems likely that the same will happen in this case as well, and that this minor victory will be short but sweet.”

One wonders if this might be a stealthy form of big (as in wholly-government pwned) ISP fish bumping off little (as in independent) ISP fish. The ISP is held responsible – it’s even spun that way in the media! – but the botnets move freely on to the next victim. And it’s all confined to Northern California providers.

Mjc • November 17, 2008 7:02 AM

I for one am quite surprised that such a massive botnet ring could be taken out so easily because of a single-point of failure – especially one hosted within the USA. However, I expect they will ‘evolve’ to use decentralised means of communication in the near future making them ever more difficult to shutdown.

Lennie • November 17, 2008 7:19 AM

@bruce The Provider was not the source of the spam, but they had customers which had mothership(s) that controlled the spambots.

@Trichinosis I don’t think so, Colo-providers only have one tool to deal with these problems. That is, if other complain about problems, they can research it, filter something, ask the customer not do it and if they persist claim breach of contract and throw the customer out.

And other providers (‘the internet’) only have one tool that is to disconnect from a network that persists to host these kind of operations.

In this and previous case the ISP’s were hosting all kinds of badness and not doing anything (or not enough) about it, even though other providers did report problems. The people that reported the problems went to the upstream-providers and told them about it and they thought it was bad enough to warrant a disconnect.

Victor Williams • November 17, 2008 7:34 AM

It’s not just that the botnet control points were on McColo, though that’s a big part of it. As said before, if the control nodes are down, the botnets can’t actually send anything.

The additional problem is that the spamvertised websites were ALSO hosted there, and without connectivity there’s no reason to send out the spam because there’s no website the victims can go to.

Sadly, the downtime was less than a week at my servers. Load is back up to previous levels.

Peter Pearson • November 17, 2008 9:37 AM

There’s a chart of the spam reporting rate at my mail service, Spamcop, posted at
http://webpages.charter.net/curryfans/peter/spamdrop.gif

The falloff is impressive, and recovery had not yet begun when the plot was grabbed.

Bob Barker • November 17, 2008 10:54 AM

The spammers will get over this pretty quickly. The ONLY thing I’ve ever see put a damper on spam was Blue Frog. Does anyone ever remember them? They were a service that automatically emailed or clicked on the unsubscribe links in any mail you marked spam. Sadly (inevitably?) they got blasted into smithereens by the bots. I think we need to go back and build something bigger and more distributed than Blue Frog to really see a true impact on the spam volume.

David Evans • November 17, 2008 11:03 AM

It’s been my earnest belief that there are weak points in the ecosystem of things like spam, phishing and a lot of other goings on. It seems to me that we often give up on attacking them because a) we think they are more resilient than they really are and b) we assume people will move onto something else equally obnoxious. I’m sympathetic to both those positions, but the alternative is collectively giving up, which seems to me to be what we’ve collectively done. It’s simply not acceptable. The problem is that governments in general and law enforcement in particular have continually demonstrated the inability to respond sensibly to Internet-based threats. Maybe over time a sort of Darwinian effect will help them to get better…

RH • November 17, 2008 11:29 AM

75% of spam…. and Storm is supposed to be sending out something like 30%.
Allowing for error in those numbers, this would indicate that Storm and McColo were responsible for nearly the entire spam gamut.

Ted Burghart • November 17, 2008 12:15 PM

Bob Barker is right, Blue Frog was by far the most effective anti-spam tool I’ve ever seen, which is why I’ve been pretty surprised that a) they folded on the first assault and b) nobody’s resurrected the approach.

Seems to me that the big ISPs, correctly approached, might be willing to support such an operation, and they could give it the infrastructure it needed to weather the attacks. When you consider the store-and-forward model of email, and the cost of managed enterprise storage, computing resources and bandwidth, I can’t imagine that supporting something like Blue Frog wouldn’t cost them less than handling who-knows-how-many billion spam emails a day.

SpamSpam • November 17, 2008 1:08 PM

This falls into the category of – Those in the best position to have the biggest impact to stop spam also have the most to profit from the existance of spam – the ISPs.

So, spam will exist until there is some external entity (i.e. regulation) forcing the ISPs to manage the spam entering the Internet through their systems/services.

The more spam there is, the more bandwidth that will be needed to carry all that extra “noise”. If there was no spam, most businesses could reduce the size of their network bandwidth connections. I know of several businesses which have had to add network capacity just to be able to handle all the spam they get each day.

Annie Nomous • November 17, 2008 2:21 PM

I also was quite impressed with Blue Security’s impact on spam, especially when I saw how little work my Frog actually did. As to giving up “too easily”, I believe the people at Blue Security were made to understand that continuing to operate the service could be very bad for their future health. This is an important consideration when dealing with criminal spammers. It’s not just your servers that they’re prepared blow off the planet.

Tree • November 17, 2008 3:13 PM

The major expense of an ISP is new infrastructure. They aren’t making money off of spam. ISPs would love for spam to magically go away, but since the cost of getting rid of spam is difficult to measure (and would require collective action, anyways) an individual company can’t do a meaningful cost benefit analysis.

me • November 17, 2008 3:20 PM

This means that all those who say nothing can’t be done are plain wrong.

It also shows that most of the spam has a single point of failure which can be shut down.

It’s amazing how just one sucker can suck life’s time out of so many people.

It should also be noted that the single point of failure of these spam operations is less technical than that irresponsible person which doesn’t care about the energy all the others have to spend to get rid of what they don’t want.

If this person can be convinced to stop, or is surrounded by others not allowing him to use their resources — which is perfectly legitimate not to allow this –, the spam can be stopped.

Nebb • November 17, 2008 3:21 PM

Just happned to come across spampoison.com today. Anyone know anything about it?

Steve • November 17, 2008 4:12 PM

@Nebb: “Just happned to come across spampoison.com today. Anyone know anything about it?”

These sort of fake address generators have been around for quite some time and I suspect they have infinitismal effect.

Read the paper to which Bruce refers in his posting “The Economics of Spam” on this blog and you’ll see that the bot nets are essentially self-correcting, purging most bogus or dead address (Section 3.3) automatically.

I suppose you could create a sort of honeypot which would act like it was accepting spam emails but then throwing them on the floor but it would have to provide a phenomenal number addresses in order have any observable effect. And even so, the spammers would soon twig to it and update their botnets to simply toss those addresses automatically.

Doug Coulter • November 17, 2008 5:30 PM

Well, I just don’t know. I have a long standing email address that gets enough spam per day to take about 3 pages to list them one line apiece. I have noticed no drop, not even for one day.

Recently my ISP started filtering for us, I bet mainly in self defense, as much of the spam was traceable to other machines on their own clueless network users. I no longer see any of that (with the few tools I have to figure out where it came from) but only 3 pages a day at one line per spam apparently from machines all over the net (presumably botnet machines) and there was zero noticeable reduction at anytime before, during, or after this story came out.

BTW, one thing they did to attract all this spam was to answer HELO and other messages to tell spammers what legal addresses were, until they finally got a clueful network admin. By then of course, the damage was already done.

I suggested to their admin that when spam comes from their own users, they ought to think of it as a business opportunity and offer to clean up that user’s machine. Their filter is pretty darn good (about 1 false detect in a year’s worth of that 3 pages per day) so it should be easy for them. They didn’t like the idea.

Just one data point, but it is one.

coljac • November 17, 2008 5:42 PM

The problem with Blue Frog wasn’t just that it had servers that could be DOS’ed or employees that could be threatened. The problem was they provided a means for spammers to scrub their lists of their subscriber’s addresses. The result was that that the spammers could find out the email addresses of Blue Frog users such as myself, and send us first, threatening emails, and then, joe-jobs out the wazoo. I had to abandon my old email address because of that fiasco – after Blue Frog blew up I was getting a spam a second.

D • November 18, 2008 12:28 PM

Why not have a spam filter that accepts the incoming message, sends it to the recipient, and then sends a false error message back to the sender saying no such address? Wouldn’t that work with the self-correcting aspect of the bot nets to purge valid emails from the lists?

tom • November 18, 2008 7:02 PM

Steve wrote: “I suppose you could create a sort of honeypot which would act like it was accepting spam emails but then throwing them on the floor but it would have to provide a phenomenal number addresses in order have any observable effect. And even so, the spammers would soon twig to it and update their botnets to simply toss those addresses automatically.”

at least on the servers i manage, this is demonstrably untrue. even before i started baiting spammers recently, my logs were showing delivery attempts to many of the same obviously bogus addresses that i was seeing 10 years ago.

it is true that botnets are getting much more sophisticated, and that many of the linux bots use real mail servers, but the majority of spam i reject still appears to come from the same old fire & forget style spam servers.

anti-spam tools are evolving rapidly as well.

my first line of defense is a greylist proxy and tarpit. this is augmented by a frequently updated honeypot list of false addresses placed in various locations where harvesters will find it. this list is automatically generated from my mail server & antispam tool logs. any attempts to deliver a message to or from a honeypot address result in immediate blacklisting. the proxy classifies hosts into one of three categories: white, grey or black.

whitelisted mail hosts simply bypass the proxy and go directly to my real mail server.

grey (unknown) mail hosts are delayed for a few seconds, then told that there was a temporary failure – try again later. those that successfully retry (a minority) are whitelisted and connected to the real mail server. however, feedback from other anti-spam tools on the real server can cause hosts to be removed from the whitelist and/or placed on the blacklist.

to blacklisted hosts the proxy behaves like an open relay, but with two important differences: it never delivers messages, and it stutters (communicates at a rate of one byte per second) causing spam servers to waste an average of 6.7 minutes trying to deliver each message. badly configured ones can stay connected for hours during which they might have been delivering tens of thousands of messages.

my users and i often go for days or weeks w/o seeing a single spam in their inbox

Seth Breidbart • November 18, 2008 7:29 PM

D, spammer don’t care if their mail bounces. Someone I know turned on a domain that hadn’t been accepting mail for five years and it was still constantly receiving spam.

Bluefrog attempted a DDoS against spammers. That had several problems: first, the bad guys are better at it (and BlueSecurity had no prevention in place). Second, how do you know your target really is a spammer? Someone could register a domain, point it at whitehouse.gov, and send spam for it. Do you really want to attempt to DDoS that domain?

After this takedown, one of the spamgangs went to an ISP in a foreign country. They lasted a few hours. (McColo and its upstreams were told about their problem years ago, they didn’t act until it was about to get embarrassing. A legitimate ISP canned the spammers in hours following notification.)

B • November 18, 2008 8:44 PM

It does make it easier to detect spam. Most people are surprised by how much spam can be caught by simple approaches based on the source network.

The problem is that the relatively small fraction of spam volume not caught by such an approach is still, in absolute terms, a huge amount of email.

John David Galt • November 21, 2008 10:50 PM

@SpamSpam: “Those in the best position to have the biggest impact to stop spam also have the most to profit from the existance of spam – the ISPs.” How is that possible? It’s true of phone service (and telemarketing) because most calls generate revenue for at least one phone company. But hardly anyone still pays by the message to send or receive e-mail. Most service is flat-rate, and thus, the fewer messages sent the better from most ISPs’ point of view.

The lessons I learn from this episode are these. First, a law forcing ISPs to charge for each message sent will NOT be a cure-all for spam; if it happens, it will turn (at least some) ISPs into the natural allies of spammers.

Second, a good approach that would work is for someone (a police agency? large ISP? public interest org/NGO?) to start sifting raw packet traffic in search of the control messages for these botnets — then shut down (or blacklist) the machine giving commands to the “bots”. Designing a firewall smart enough to block just those packets would also be helpful, especially for all the people running unpatched, common operating systems who are hosting bots without knowing about them.

Schneier on Security

Most Spam Came from a Single Web Hosting Firm

Comments

Leave a comment Cancel reply