Schneier on Security
A blog covering security and security technology.
« Smart Profiling from the DHS |
| El Al Doesn't Trust the TSA »
May 23, 2006
Spammers Win One
Blue Security was an Israeli company that fought spam with spam:
Eran Reshef had an idea in the battle against spam e-mail that seemed to be working: he fought spam with spam. Today, he'll give up the fight.
Reshef's Silicon Valley company, Blue Security Inc., simply asked the spammers to stop sending junk e-mail to his clients. But because those sort of requests tend to be ignored, Blue Security took them to a new level: it bombarded the spammers with requests from all 522,000 of its customers at the same time.
That led to a flood of Internet traffic so heavy that it disrupted the spammers' ability to send e-mails to other victims -- a crippling effect that caused a handful of known spammers to comply with the requests.
Then, earlier this month, a Russia-based spammer counterattacked, Reshef said. Using tens of thousands of hijacked computers, the spammer flooded Blue Security with so much Internet traffic that it blocked legitimate visitors from going to Bluesecurity.com, as well as to other Web sites. The spammer also sent another message: Cease operations or Blue Security customers will soon find themselves targeted with virus-filled attacks.
Last week Blue Security gave up:
Wednesday, Blue Security said it had to give up because it couldn't sustain the fight against spammers. "Several leading spammers viewed [us] as a strategic threat to their spam business," Eran Reshef, Blue Security chief executive wrote in the message posted to the company's site.
"After recovering from the attack, we determined that once we reactivated the Blue Community, spammers would resume their attacks. We cannot take the responsibility for an ever-escalating cyber war through our continued operations.
"As much as it saddens us, we believe this is the responsible thing to do," said Reshef, who did not respond to an e-mail requesting additional comment. Later Wednesday, a spokesman said that the company would not be making any additional statements beyond the message on its site.
Another news article. And Marcus Ranum on Blue Security's idea.
Posted on May 23, 2006 at 12:58 PM
• 40 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
i saw this elsewhere, metafilter i think. damn, they folded.
didn't lycos briefly offer a little package that would shoot back at popups?
it was probably a mistake to implement this with a centralized corporate front ("target") leading the way. if it were noncentralized and organized only at the level of individual users, the spammers would indeed have to identify and target everyone, which is harder to do.
against email whitelisters, it's impossible. the junk folder doesn't even count toward my 25 mb storage limit in hotmail, which should consider adding this feature. let's see if these mothers can take down microsoft!
Eventually, the spam situation will come down to either vigilante justice (http://tinyurl.com/ryury) or the re-invention of email. Personally, I'm rooting for the latter, though it's hard to get too choked up about the former.
I'd argue (like Marcus does, essentially) that Blue Security is not in fact fighting spam with spam, but instead fighting spam by enabling people to effectively complain about spam. A complaint isn't a "spam" (IMO), since the complaints are responding to an attempt to solicit business, and ergo are not themselves unsolicited (unwelcome, maybe, but certainly solicited!)
I think we already have several different versions of re-invention of email.
One of my private addresses, for example, (which is used by a very small number of people) runs behind a procmail recepie that checks origination email addresses against a database of "okay" email addresses. Family members get through, everything else (including messages from "myself") gets roundfiled. That address gets no spam, but it is a totally nonscalable solution... it works because it's a private address, and I'm willing to give up a lot of the upside of mail to have it work the way I want it to...
Many email servers now implement delay-first systems, or FQDN checks, or other methods to cut down on obvious bulk mail senders.
Blue Security was innovative in that they weren't attempting to develop a new filtering or mail handling methodology for dealing with spam but instead attempting to make it economically infeasible for spammers to send spam.
It's too bad they don't have the market presence to enable a technological footprint that could handle PharmaMaster's assault. However, the mere fact that the service caused such a reaction on the part of the spammers leads me to believe that the spammers, at least, weren't happy with the business model.
Here's a company that Google should buy :)
I pretty much lost any respect I had for Blue Security when they "handled" their DDOS by shifting it all to Six Apart. Blue Security knew what the possible response to their tactics would be; "forward DDOS traffic to innocent bystander" is not an acceptable backup plan.
@Roy, I don't think either vigilante justice or email re-invention is required --- just proper targeting of law enforcement.
The problem with the current law enforcement approach to spam is that it targets spammers -- the people who make technical arrangements for spam to be sent -- rather than advertisers -- the people trying to sell stuff.
Spammers are notoriously hard to track down, their methods of concealment being legion. Advertisers, on the other hand, are connected to their customers by money transfers, which leaves a trail that can be followed by standard police methods.
The police should set up frequent sting operations, responding to spam offers with ostensible purchases, tracking where the money goes, and busting the people actually peddling the pills/stock tips/loans.
Eventually, the advertisers would conclude that the risk is not worth the reward, and would cease contracting out their advertising to the spammers, cutting off their air supply. After which we can all give SpamAssassin a well-deserved rest.
An error in judgement during a crisis situation doesn't invalidate your business model or design. Don't base your respect for the technology on your respect for the management decisions.
Although, I agree, it was magnificently dumb...
@ Carlo, Roy
Remember that proper targeting by law enforcement requires jurisdiction (and resources, etc., but without jurisdiction you've got nowhere to go even if you have the money to do so).
Since the Internet is by and large wild and wooly to begin with, and international to boot, the only really effective way to establish jurisdiction is to push it onto the maintainers of the network (commercial entities) instead of governmental entities (that are usually tied to a physical location).
And that, I think, is a bad idea. As nice as it would be to know that there was a central networking complaint management system whereby AT&T, WorldCom, etc. would simply cut off all traffic from spammers when reported, such an entity would be enabled with far too much authority for me to be comfortable...
The other part of the Blue Security model that I loved was that they maintained a list of opted-out email addresses and tools that would let legitimate email marketers clean their lists.
And, yes, one unsolicited email => one opt out request was the maximum response but it scaled up to that. The first response on an unsolicited email was just a few opt-out requests with a pointer to Blue's list cleaning tools. If the senders didn't comply by cleaning their lists, the response would scale up to one opt out per email received. When the community is hundreds of thousands and they all got the same email, the spammer got a lot of opt out requests. :-) I loved it. It appealed to my sense of justice.
Now, with the removal of Blue's human reviewers, there can be no scaled approach. If the open source community takes this up (as I would love to see happen), it will be one response per received email from the start with no easy list cleaning option.
"it was probably a mistake to implement this with a centralized corporate front ("target") leading the way. if it were noncentralized and organized only at the level of individual users, the spammers would indeed have to identify and target everyone, which is harder to do."
Centralization is necessary to make sure that false spam reports were not generated. If there was not a trusted, centralized group to verify spam complaints, then there will be nothing to prevent the system from being taken over false complaints which would lead to DDOS attacks.
The point I was trying to make is that while this is an Internet problem (with the International jurisdiction issues that you mention), it is amenable to a solution that is entirely independent of the Internet.
No burden of any kind need be placed on ISPs, telcos, or carriers of any kind. What is required is that a few *vendors* of spamvertised products should be busted.
If the Viagra bootleggers get to wondering whether the any of the orders that came in after they spamvertised are actually from a police agency instead of from a bona-fide erectile dysfunction sufferer, that's going to cause them to reconsider their contract with the spammers.
The point is, the vendors are the root of the problem, and they are easily traced. The spammers are just independent contractors, and they are hard to locate. So we should squeeze the vendors, and the spammers will dry up and blow away of their own accord.
I think we have to go International in the Internet Law and Police, since it's so easy to open up a Russian (or wherever there is no internet law) web site that goes and help or use spammers.
Another method to suppress the companies using spammers would be to use crackers to infiltrate the pharma sites and find a way to grab all the money being spent there in blackmail so that they stop using spammers. If they don't, we squeeze all paiements away from their sites. With no money coming in, they'll stop !
Plenty of other methods could be used to attack pharma sites known to use spammers that would force them to stop.
One question comes to my mind, who orders from these pharma sites since we know that most of their products are useless/fakes ?
It's not just the the spammers and the vendors, but also the end users. The spammers and vendors are just capitalizing on morons that actually respond to broken Engrish emails advertising "C / A L L / S" or "Super P_1_L_L" or foreign lotteries or "Low h0me l0an rat:es" or free stuff. If no one actually responded to these things, there would be no vendor or spammer trying to slip trash into your inbox.
I am surprised that anyone thought the Blue Security approach had any merit at all, and that's completely ignoring the question of the legitimacy of fighting spam with spam.
Maybe things have changed in the past couple of years. In my experience before then, in most spam a return address or an opt-out address was one of two things: either (1) a completely bogus address, in which case Blue Security would be pounding some hapless network or postmaster with a lot of useless unsubscribe requests, or (2) a destination used to gather up valid email addresses of people who were willing at least to open a spam message and respond in some way. Lists of such addresses were a salable asset to a spammer. Hence the advice always was, and I think should still be, _never_ respond to a spammer. And questions of vigilantism aside, I can't see under these circumstances that the Blue Security approach to spam could ever have done much good.
I do respond to abuse addresses via spamcop (and on really rare occasions directly to the appropriate abuse addresses). But I don't respond to a spammer.
Sounds vaguely like an Internet Terrorist attack.
Who is in control of the Internet?
I have to agree with J to some extent. How is this really effective with spam as a whole? I see more innocents getting bombarded with these emails than the real cause. I get tons of auto-responses from mail servers saying an email was not delivered for one reason or another. I never send those emails. They never originated from my server. And the origination had nothing to do with the MX records for the domain they claimed to come from. I'm almost as sick of the mailer daemon errors as I am of spam itself. Of course it doesn' help that I set a default delivery address for the domain... but there's good use for that, providing the stupid bounces from spam not sent by me weren't coming in.
As for my protection against spam... I go with an approach that can only work in personal email servers. I block subnets that are known for delivering email. China, Russia, and most of South America can't even connect to my port 25. I don't know anyone from those regions of the world, and I can accept not getting to know somebody from those regions of the world (through email at least). It cut my spam down by about 80-90%. Canada is now top on the list, but much harder to block by subnet due to overlaps with US subnets.
oops... "known for delivering email spam"... not just delivering email. heh.
The only surprise here is how long it the bad guys to exploit this pre-packaged public DDOS server. Good riddance to a terrible idea.
j, Alex S - you misunderstand the approach that Blue Security took.
That was the point of the human reviewers - they would manually examine the website of the advertisers, and determine the most appropriate way to send a complaint message.
They knew not to reply to the 'from' address of the spam - they weren't dumb, these folks.
@ Carlo Graziani
Interesting solution. However, how do you stop someone from attacking the competition by advertising a competitor's product or website through the spam? With the mass emails effectively costing pennies, one could advertise their competitor's product, the competitor will get a temporary increase in business for reasons unknown to them, and eventually will have to answer to law enforcement, and probably pay a hefty fine or be shut down.
Then, following that, how do you prove that they did or didn't contract with the spammer to advertise the product?--especially given that, as you said, spammers are notoriously difficult to track down, you would have no proof whether or not the company was actually the one who initiated the spamming.
Not that it would necessarily happen, but that is a consideration. There is no authentication here that I see.
people seem to think bluesecurity was the good guys. they weren't. they were greyhats that were attacking spammers, vigilante style.
I didn't use the Blue Security service, but if they followed the protocol they were supposedly using, I wouldn't call what they were doing either vigilantism or grey-hat behavior.
If someone opens an unsolicited marketing communications with you, you are entitled both to ask them to cease and to have that request honored (this is US legal precedent, not necessarily a moral judgement, although I happen to personally think we have a right to opt-out of communications that are unsolicited).
Nothing in Blue Security's business model is outside of that procedure.
There is an open source project to take up the reins of where BS left off. Hopefully without the centralized infrastructure that forced BS to blink first.
their buisness model was to 'overwhelm' (their phrase) the spammer's mail servers. that is the aspect I object to.
Crap, that last one was by me, and '@Pat Cahalan' was supposed to be the first line, my bad.
My take on this is that this reaction is an indication of effectiveness. The only surprise is Blue Security's limp fish response. The right response is to weather the storm - hold out and starting working with their ISPs to block the incoming traffic. So what if the botnet is 15,000 or 100,000 machines? It can't stay up forever. Eventually people would notice, machines would be taken down, and more effort would be required by the spammers to keep up the attack. Trace the machines and get them shut down. Work the phones. Get someone who speaks Russian on your side and track down the botnet controllers. You can't possibly hide something of that magnitude for long, and eventually the spammers will piss off enough people that they won't be able to maintain the DDoS.
Who cares if the Blue Security site is down for a few weeks? Anything that gets them out of their holes is a good thing. They quit just when they were starting to win!
"their buisness model was to 'overwhelm' (their phrase) the spammer's mail servers."
Not specifically mail servers, but instead the servers that actually complete transactions initiated by the spam, whether mail or web form or what. And by sending one response for each spam sent out. Do you think it's unreasonable for a business to be forced by a company like Blue to have the server capacity to handle one request per spam solicitation sent?
Come to think of it, I wonder how Blue Security handled businesses that, on the initial contact, denied having contracted the spam. Would they be in the position of having to decide which companies really didn't, and which companies were just trying to avoid getting on Blue's spam list?
Pretty much any system of "punish the spammers" is vulnerable to abuse like that.
Do you really think it's that hard to tell the difference between a legitimate website and a spam-advertised site? Their method was cautious and careful and certainly accounted for there being doubt about a particular site. If there was doubt on the part of Blue Security, simply leave the site off the list and re-visit it a few times. Time was on Blue's side. Sending spam does cost some money, eventually. Switching sites and dns names costs time and money. If spammers were sending spam to attack competitor's sites or innocent sites, they are losing money. Eventually either the spam would stop or it would become clear the site was using spam advertising and should be included in the complaints.
"If spammers were sending spam to attack competitor's sites or innocent sites, they are losing money."
Without any people trying to punish vendors who use spam to advertise, it's true that it's generally a money-losing proposition. But if you try to get law enforcement, or companies like Blue Security, involved, all of a sudden Joe Jobs can cause a really major hassle for the target.
These are good points, and the Devil is in the details as usual. I would
say that the police have a certain evidentiary burden to meet, and would
certainly have to plan to counter the "I Was Framed" defense in court.
So what would be required would be a full-up undercover investigation,
not a mere perfunctory "you filled my order, you're guilty". Bank records,
documenting payments to spammers. Possibly sting operations, posing as
spammers to attract business from advertisers. Electronic surveillance
(presumably court-sanctioned, sigh...). You know, gumshoe stuff.
The point is, focusing on the vendors opens up a lot of possibilities for
traditional police techniques, possibilities that don't exist when
targeting the spammers themselves. Naturally, this doesn't relieve the
cops from their obligation to actually meet the burden of proof in court.
@pdf23ds: Avoiding Joe Jobs and other spammer tricks is exactly the service Blue Security was providing. They went after the website advertised in the spam mail. The weakness of spam is that at some point you have to extract money from your spam-suckers, and that means a functioning website.
Here's how it would work: the anti-spam company contacts the site because they have been receiving spam directing spam-suckers to the site. The site has two choices: admit or deny. If they admit, the choices are to suffer a complaint storm or stop spamming the anti-spam company's customers.
If they deny, there are two choices - either they really are innocent, or they are lying. If they are innocent, then the spammer is essentially advertising their site for free, which costs the spammer money. That can't go on indefinitely, since spamming does in fact cost money.
If they are lying, there's another two possibilities: either they remain where they are and eventually it will become apparent (after much time has passed) that they are in fact advertising via spam, OR they will be forced to move the site to avoid being branded a spam-vertiser. Either solution is not palatable from the stand-point of the spam-vertiser, since it increases costs and puts pressure on them.
None of these solutions is attractive for the spammer. Their margins are thin as it is, and anything that introduces infrastructure upgrades, hassles, and essentially any human intervention will kill their business ... which is the whole point of the exercise. A hostile environment for spammers is what we want, an uphill slope. You'll never be able to get rid of it, but you can cut the volume.
There are at least three very interesting aspects to this.
One is the number of posts claiming that Blue Security's activities were unethical or otherwise questionable. In some forums, some of these actually seem to have originated from spammer propaganda, others seem to be people fooled by the spammers. Of course such a claim is somewhat ad hominem, and doesn't invalidate their points of itself; but in fact, every moral argument raised against them was actually already met by the way the company went about its operations .
In particular, they did NOTHING to actual spammers; all the complaints were directed to the companies paying the spammers . While this was nothing like a flood attack (these sites were only getting a few thousand complaints spread across days, which even a 486 on a dial up could easily handle), it was an effective economic disincentive to abusing a public resource for customer acquisition (spamming) because it increases the marginal cost (of customer acquisition through spamming) to something closer to its true cost.
It is also a strong psychological disincentive, because it effectively tells the spamvertising company: here are thousands of people who were potential customers (at least according to your "business partners") who, because of your behaviour, will now never do business with you. In other words, if a spamvertiser was "semi-legit", i.e. had legitimate products or services but marketed them unethically, then Blue Frog was helping to send the message that if you spamvertise, you are locking in a limit to the growth potential of your business. You can never get really big. You can never get really wealthy.
The nett result was that spamvertising companies were giving less money to the mob, and thus taking away the powerful economic incentives which had so far led them to get around every possible technical barrier that had been put in place. In this sense, Blue Frog's approach was not only the right approach, it was the only approach so far that was really working.
This highlights the second interesting point: a company wasn't just forced out of business by a massive DDoS, they also had their DNS altered (current rumour is that this part appears to be an inside job), and then hundreds of posts on various internet forums claiming that it was OK and they deserved it. Organised crime is now heavily involved in spamming (and host of closely related activity), and they are making an absolute bundle out of it. Thus, they will resist ANY countermeasure that works, whether it be technical, process, economic, social, legal, or whatever. They will use any resource available to them in order to do so. That includes propaganda, law suits, technical countermeasures, technical counterattacks, bribery of technicians, bribery of journalists, bribery of law enforcement officials or judges, bribery of politicians, veiled threats of violence, actual violence, and murder. And that doesn't just apply to spam; if any computer security measure starts cutting into their profits, then they are going to work hard to stop it. And they are big, well organised, very wealthy, and very experienced at winning this game.
So if you are going to work to improve internet security, bear in mind that you are going to have to fight to do so, against vested interests who do not want it to be secure, and can bring considerable resources to bear to make sure of it.
Which brings me to the third point: an surprisingly large number of people are now calling for vigilante justice. There are two basic flavours: those who say that it is time to start proactively destroying botnets by disabling infected machines wherever you find them; and those who are talking about collecting money to have "PharmaMaster" terminated. No, they don't mean his operations shut down, they mean really dead, bullet in the brain dead. And I'm not hearing it from one lone loon who is promptly shouted down, there seem to be a lot of people suggesting it and the replies are mainly about overcoming technical difficulties (e.g. securing anonymous payment to the killer). I certainly don't support that view, but if nothing else, it indicates the extreme level of outrage that PharmaMaster has engendered. (Who knows; Lucky Luciano had Dutch Schultz whacked because he was a "loose cannon" who was causing too much public outrage. Maybe "PharmaMaster's" bosses will do the job for us?) This is not simply outrage that PharmaMaster would dare to shut down a security business through DDoS, it is outrage that the power of crime on the Internet has become so great that they can afford to, and dare to, very publicly destroy anything that seems to threaten them. The impression created by doing this so publicly is that we have indeed reached a tipping point (the "Capone-Chicago of cyber fraud" to slightly misquote Bruce Sterling), where if they aren't stopped soon, by any possible means, then they will be impossible to stop because they will be running the whole show.
The other vigilante view, the destruction of botnets, I have to admit I am seriously considering. The two main counter-arguments are increasingly less credible. Firstly those who say that law enforcement needs to be left to law enforcement officials are right, but what do you do when law enforcement has been so clearly, utterly and totally impotent for so long? Traditional law enforcement on the internet has been very close to completely and utterly useless. Sure, they catch lots of kids doing petty vandalism and then slap them with viciously Draconian penalties. But very few serious criminals are caught. Consider bot herding; to anyone who has bene following internet security, it is pretty clear that shutting down bot herds are our numbers one, two and three priorities. But police have so far caught, what, six bot herders altogether? And after deliberation over a period, decided that even in those cases they could not only not shut down the bot nets, but could not even send warnings to the machines' owners. This is pathetic. If law enforcement in a physical town was this weak, you'd take it for granted that they are on the take.
The second issue is the effect on the legitimate owner of the bot host; isn't he an innocent bystander? It is getting to the point were a lot of people are saying, to hell with that guy. The harm he is unwittingly causing is far greater than the harm caused to him by torching his polluted OS, and in any case there is a high likelihood that he will himself suffer serious financial losses (e.g. through keylogging) if the bot on his machine isn't stopped. At the risk of using a real world analogy, no one would blame you for kicking down your neighbour's door and hosing down his front room if you saw flames through the window. It probably isn't necessary to format the drive or anything nasty like that; just rename the NIC driver to "I'll do it again, and worse, unless you disinfect and harden this machine before reconnecting", put an explanatory text file in every writable "Start Up" directory, and reboot .
1. A corollary to point one is that Blue Frog was vulnerable to DDoS for the very reason that they exercised this centralised control to ensure that operations remained ethical. Many people have since suggested a decentralised, open source version instead. But so far all such models *are* vulnerable to the sort of complaints that have been wrongly raised against Blue Frog, e.g. risk of joe jobbing. It will be a significant challenge to build a resilient, decentralised system that still offers the same checks and balances which Blue Frog provided.
2. A point of clarification here for those who don't understand the industry: spam is sent on behalf of businesses, who range from the legitimate but ethically bankrupt, through to the outright criminal, but these businesses rarely have anything to do with the actual spamming process. The spamming is mainly (although not exclusively) sent by bot herders, who are criminals who have taken remote control of very large numbers of innocent (but careless) people's home computers. These bot herders don't much care what their botnets are doing so long as it earns money, and when they are not sending spam they may be sending phishing email, sending stock manipulation email, anonymously relaying illegal traffic through "undernets", or running DDoS protection rackets (mainly against dubious off-shore casinos and porn sites who will just join the fold and not try to involve the cops). The spamvertised companies (who, as we mentioned, are often quasi-legit) usually don't have any direct contact to the bot herders, but contact them through various legitimate-seeming fronts which are operated by organised crime. In the US, the Gambino family has a major involvement in this area, with the technical side of the operations being concentrated in Florida. Elsewhere there is major involvement in eastern Europe controlled or influenced by the Russian mob. China is another significant area but I have no information about how the business is organised over there.
3. According to my count, two in California, one in the UK, and three in the Netherlands. The 3 in the Netherlands may or may not have been big timers (reports differ on how many bots they controlled) and were using them for a protection racket. The other three were all small fry and were using their botnets for click-thru fraud, which a) makes it easier to track down the perpetrator (he's the one getting way too much money for his crummy site) and b) means the victim is a company, who can afford to involve private investigators.
4. Yes, I am assuming it is a Windows machine.
Your analysis seems pretty good. But I think you underestimate the extra room that spamming vendors get by denying to Blue that they're spamming.
"If they are lying, there's another two possibilities: either they remain where they are and eventually it will become apparent (after much time has passed) that they are in fact advertising via spam, OR they will be forced to move the site to avoid being branded a spam-vertiser."
That's what happens when Blue trusts the vendor's denial by default. But that "much time has passed" is plenty of time to make a profit, unless Blue is more agressive about investigating denials.
If Blue lowers its standards, and starts putting vendors who deny spamming on the "bad list", it makes it easy enough for spammers to frame legitimate businesses that they'll start doing it, and when this framing or Joe Jobbing starts going on at a large scale, it becomes actually necessary for Blue to put a lot of resources into investigating each claim, because there now *is* a good chance that the vendor's denial is truthful.
Stop spammers? easy, find 5 (especially the russian), take them to a very public event (say the superbowl) then execute them - no fuss, no questions just executed then broadcast a message saying that any other spammers caught will get the same. Ah but cant you just hear the liberals screaming
@Sorry this is so long winded
Thank you very much for your excellent and clearly stated explanation of Blue Frog's approach and the surrounding economics.
I would simply have made the observation that 100% of the people who complained about Blue Security and called their operations a DDoS program, have been too intellectually lazy to take the time to understand what they were doing.
Of course, the fact that I wouldn't have bothered to write up an explanation as careful as yours doesn't say a whole lot for my own intellectual gumption, does it?
plz could you relate me to some crackers emails?
Looks like the Blue Frog guys are back from the dead:
Interesting change -> same technology, new target, and one that can't retaliate the way the spammers did.
Schneier.com is a personal website. Opinions expressed are not necessarily those of Co3 Systems, Inc.