Spam as a Business

Interesting research: Kirill Levchenko, et al. (2010), “Click Trajectories—End-to-End Analysis of the Spam Value Chain,” IEEE Symposium on Security and Privacy 2011, Oakland, California, 24 May 2011.

Abstract: Spam-based advertising is a business. While it has engendered both widespread antipathy and a multi-billion dollar anti-spam industry, it continues to exist because it fuels a profitable enterprise. We lack, however, a solid understanding of this enterprise’s full structure, and thus most anti-spam interventions focus on only one facet of the overall spam value chain (e.g., spam filtering, URL blacklisting, site takedown). In this paper we present a holistic analysis that quantifies the full set of resources employed to monetize spam email—including naming, hosting, payment and fulfillment—using extensive measurements of three months of diverse spam data, broad crawling of naming and hosting infrastructures, and over 100 purchases from spam-advertised sites. We relate these resources to the organizations who administer them and then use this data to characterize the relative prospects for defensive interventions at each link in the spam value chain. In particular, we provide the first strong evidence of payment bottlenecks in the spam value chain; 95% of spam-advertised pharmaceutical, replica and software products are monetized using merchant services from just a handful of banks.

It’s a surprisingly small handful of banks:

All told, they saw 13 banks handling 95% of the 76 orders for which they received transaction information. (Only one U.S. bank was seen settling spam transactions: Wells Fargo.) But just three banks handled the majority of transactions: Azerigazbank in Azerbaijan, DnB NOR in Latvia (although the bank is headquartered in Norway), and St. Kitts-Nevis-Anguilla National Bank in the Caribbean. In addition, “most herbal and replica purchases cleared through the same bank in St. Kitts, … while most pharmaceutical affiliate programs used two banks (in Azerbaijan and Latvia), and software was handled entirely by two banks (in Latvia and Russia),” they said.

This points to a fruitful avenue to reduce spam: go after the banks.

Here’s an older paper on the economics of spam.

Tags: , , ,

Posted on June 9, 2011 at 1:53 PM • 37 Comments

Comments

Chris Kanich • June 9, 2011 2:11 PM

I’m an author on this paper (as well as the older one) and would be happy to answer questions in the comments here. This work is the product of a lot of effort by a lot of people and I’m glad it’s gotten the reception it has so far.

John Perich • June 9, 2011 2:39 PM

Fruitful, perhaps, but how likely to be implemented? Going after a bank means (indirectly) going after all of a bank’s customers. Do you think St. Kitts might have a few U.S. customers who’d object to international investigation?

bcs • June 9, 2011 2:51 PM

What I’d like to see done is to levy a tax on spam such that it doesn’t matter who pays it. From there you could define a liability chain starting with someone/something that is easy to find (banks, the end merchant, etc.) and ending with you guy you really want to catch. In the middle you arrange things so each link can be expected to (given the right motivation) be able to identify the next. Then when a spamming goes out you can go to the first link any tell them: “pay the tax, or rat out someone further down the chain”

Mark Boss • June 9, 2011 3:27 PM

To me, the crazy thing about spam is that it works. If it didn’t make money, people wouldn’t bother making spam. Much like illegal drugs. Drugs continue to sell because people keep using them. I’m not sure a war on spam will be much more effective than a war on drugs.

vwm • June 9, 2011 4:06 PM

Mark, it does not actually need to work. It’s sufficient if enough people think, it works:

Consider I started spamming today, since I read the “old paper” and now I think it’s a great opportunity. I can easily inconvenience a tremendous amount of people, before finding out that it’s useless (or getting caught). Meanwhile, a tremendous amount of people is lured into thinking, it pays: “Because, you know, like, otherwise that other guy would not do it”.

Paul Crowley • June 9, 2011 4:53 PM

There’s clearly only a handful of banks which are the best suited to spammer purposes. However, we don’t know yet whether other banks are much less suited, or just a little less suited. We’ll find out only when we succeed in reducing the spam transactions on one of them – to what extent will spam go down, and to what extent will another bank pick up business as a result?

Impossibly Stupid • June 9, 2011 5:01 PM

What vwm said. This research is flawed by thinking that the sales transaction matters at all in the economics of spam. If you want to look into shutting down banks, try searching not for the ones that process orders, but for the ones that those banks send money to in order to pay for spamming services.

Stefan Savage • June 9, 2011 5:07 PM

Paul, the key issue isn’t the number of banks per se (although it plays a role) but rather the value of those relationships. Bank relationships are expensive to create (in both time and capital) and thus expensive to lose. Not infinitely expensive — there is turnover among these banks over time — but much more expensive than other resources (e.g., botnets, bp hosting, domains, etc). The potential level here is time. If an organizations starts using a new acquiring bank, this is immediately reflected in the payment transaction. If big US issuers decided to not to settle with particular merchants and categories code for such banks, the are positioned to act many orders of magnitude more quickly than new acquiring bank support can be obtained. Usually in security the asymmetry supports the adversary… this is one of those cases where it supports the defender. Now as to whether there are proper incentives to get US issuers to consider this intervention is another question… and not one that we’re in a position to answer just yet.

Stefan Savage • June 9, 2011 5:15 PM

Hi Impossibly Stupid,
Ultimately the sales transaction does matter to the economics of spam, because this is the source of all the funding in the spam ecosystem. Affiliate commissions, botnets, bp hosting, bulk registration, etc all are powered by consumer payments… there’s no other subsidy.

Moreover, as a rule, payments to those who provide individual services (e.g., spamming) is not done via credit card or even direct bank transfer and instead will occur out of band (typically through yet another organization) via an online money transfer service such as WebMoney. This is not easy to track.

tommy • June 9, 2011 5:23 PM

Mark Boss made a good analogy, but did not follow with the logical conclusion: Legalizing drugs solves a whole host of problems, following perfectly the prohibition of alcohol in the US between 1920 and 1933, when it was realized that it supported huge criminal enterprises, corrupted the LE and Judiciary, and there was massive disobedience by otherwise law-abiding citizens.

Now we have an additional benefit or two: A huge drop in incentive for spammers, when I can buy the same thing legally, locally, and of known purity from a brand-name manufacturer; huge savings to consumers; and the assurance that it’s the real thing, as opposed to counterfeit drugs or those that have been cut, sometimes with toxic substances.

Replicas that are actually counterfeit (i. e., bear the name “Rolex” vs. “Romex” or something): Go after the manufacturer and everyone along the chain, for illegal use of a trademark.

I don’t think that stopping spam 100% will stop the sale of pirated software, but certainly will make a dent if the pirates can’t advertise their product quite so freely. Much good, and no harm, from going after the banks, presumably by international task force.

Richard Steven Hack • June 9, 2011 6:15 PM

Put me down on the side of those who think that even if you manage to find leverage against every link in the spam chain – it still isn’t going to stop spam completely.

Now whether one can put enough of a DENT into it that 95 percent of all email is no longer spam might be worth the effort – or not.

There needs to be a cost benefit analysis of how much it would take in resources to reduce spam by how much of the percentage of email it might actually achieve.

But in the end, it’s highly unlikely to be able to stop one hundred percent of spam.

One problem with going after the few banks currently involved is that even if they cut off the spammers, some other shady bank will move in to take over – IF there’s enough money in spam. IF there’s enough money in spam, I wouldn’t be surprised to see 1) banks started BY spammers, and 2) other means of payment being created by spammers. 1) might be able to be handled by the same means as removing the first banks, but the point is it’s likely other banks will be created. 2) might not be able to be dealt with at all depending on the nature of the mechanisms.

It’s similar to the problem of sanctioning Iran. Iran is able to evade most of the sanctions because there is a business motivation for many financial institutions in various countries to maintain business ties with Iran. Also, there are mechanisms in the Muslim financial world to evade Western sanctions.

The same sort of thing could exist or arise in the case of spam, albeit on a smaller scale. The main difference is that Iran is liked (or viewed as useful) by more people than spammers are. But given the level of spamming, clearly some people like money more than they hate spammers. This problem won’t go away.

Bottom line: It may be worth a shot, but don’t expect complete success. Like all organized crime, the motivation and adaptability of the criminals are probably greater than the ability of law enforcement to impede them.

Taking out X number of spammers or their banks is like taking out X number of drug cartels. You still end up with a drug problem. As William S. Burroughs used to argue, the drug problem is a pyramid with the users at the bottom fueling the entire pyramid. In the case of spam, the people actually buying spammer product are at the base of the pyramid. Unless you do something about them, you really can’t solve the spam problem.

Spam in a sense is another example of what happens when stupid humans meet technology. You have a problem that can’t be dealt with completely because it involves human nature. Without altering human nature, the problem cannot be solved completely.

Stefan Savage • June 9, 2011 8:34 PM

Hi Richard,
Stopping spam can’t be the rational goal. Its never going to be a cost effective outcome. Moreover, spam gets used for purposes that have nothing to do with direct advertising (and hence there is no direct payment bottleneck).

The way to think about this (and I think many security problems) is what is the most cost effective way to reduce the problem to an acceptable level? Our premise has been that its difficult to determine the most efficient intervention unless you know how your intervention impacts the cost structure of the adversary — if you degrade their ability to use resource X, how does that impact their profitability? How many alternatives are there and what is the switching cost? For much of spam, the payment component appears to be the weakest link with the highest switching cost and thus is potentially a highly efficient intervention.

Jay • June 9, 2011 9:01 PM

@Stefan: Until the next stock spam, or religious spam, or… Not all spam has ever been for an on-line payoff.

Jonathan Wilson • June 9, 2011 9:16 PM

Since the credit card companies (Visa and MasterCard) have large operations in the USA, why not put pressure on those companies to deny the spammers access to the global Visa and MasterCard payment networks, i.e. revoke their merchant accounts.

Force the credit card companies to insert language into their merchant agreements stating that these merchant accounts cant be used for illegal activities (which would include selling prescription drugs to anyone without a valid prescription for that drug, selling pirated copies of computer software and selling fake knock-offs of designer goods like watches and handbags) and force the credit card companies to enforce those rules and shut down merchants selling this illegal stuff.

Stefan Savage • June 9, 2011 10:43 PM

Jonathan,
A challenge is that intellectual property violations (which is what much of this is) are not illegal in a uniform fashion. For example, drugs go off patent in different locations at different times. Brand/trademark protection differs from place to place. Visa/MC are not in a position to enforce US law on foreign jurisdictions. Indeed, from the standpoint of many spammers what they are doing is advertising goods at prices for which there is demand, and attempts to stop them are considered an instrument of Western protectionism. This point of view is viewed sympathetically in a number of countries in which such banks operate, FWIW.

I think the issuers are in a better position to intervene since they operate in the jurisdiction in which the activity is criminalized.

Impossibly Stupid • June 9, 2011 11:55 PM

@Stefan Savage:
“Ultimately the sales transaction does matter to the economics of spam, because this is the source of all the funding in the spam ecosystem.”

A claim you make unsupported. Your analysis seems to be of what the “visible” economics are, not what is happening behind the scenes. Unless you try to do a spam run yourself, you have no idea how (or even if) the whole effort is worth the cost to you. The people you’d have to hire to provide those email blasts (and other supporting services), yes, would need to see a net gain on what they do, but as the person who is pushing the pills (or watches or whatever), you’re spending the money in the mere hope you’ll see a profit. If you don’t, nobody cares, so long as there is another poor sucker who comes along that also thinks they could be getting rich by pushing pills. Rinse and repeat. The vast majority of spam could actually result in a net loss; I would love to see research on that side of the fence.

“Stopping spam can’t be the rational goal. Its never going to be a cost effective outcome.”

On the contrary. It is trivially cheap and easy to stop spam of all kinds. I’ve effectively done it myself, to the point where I no longer even run filtering software on my server. Considering that I used to get 5000 spam a day back in 2004 (and actually had an account on spamcop.net shut down because of the volume), it was a monumental win. All it takes is treating email like an edge network instead of a node network, the end solution being disposable email addresses.

Jorrit • June 10, 2011 12:44 AM

@Chris Kanich:

Here’s a question for you: do you think there’s a chance that part of the spam industry is funded (secretly, of course) by anti-spam companies?

Daniel • June 10, 2011 1:04 AM

Is there are research that tries to come up with a reasonable cost of the impact of spam? I’ve always treated spam as more of a personal hassle than anything else. The “cost” of being a drug addict is obvious to anyone who has been around one. The cost of spam seems more abstract, less personal.

tommy • June 10, 2011 2:22 AM

@ Daniel:

“The cost of spam seems more abstract, less personal.”

Yes, because you can’t see it right in front of you, as you can an alcoholic driver. There’s a decent artice at WP, although I haven’t looked at all the citations:

http://en.wikipedia.org/wiki/Spam_(e-mail)

Quick, pertinent excerpts:

“ISPs have attempted to recover the cost of spam through lawsuits against spammers, although they have been mostly unsuccessful in collecting damages despite winning in court.”

— Which therefore means that some part of your monthly bill for Internet service goes to pay for the bandwidth and other resources used by spammers. Not to mention slowing the network with this junk traffic, which is a mild, if unintended, Denial of Service attack.

“According to the Message Anti-Abuse Working Group, the amount of spam email was between 88–92% of email messages sent in the first half of 2010.”

“According to a Commtouch report in the first quarter of 2010, there are “…183 billion spam messages” sent every day.”

Again, think of the load on the Net infrastructure, ISPs, etc.

Dollar cost, from one of the citations at that article:

“Analyst Richi Jennings projects the cost of spam in 2005 will come to $17 billion in the United States and $50 billion worldwide. These figures reflect the productivity loss to the diminishing number of business users without spam filters, the cost to purchase and administer anti-spam systems, and time wasted dealing with spam that gets through and with legitimate messages that have been misidentified as spam.”

Not to mention the suckers who get ripped off or bot-netted, speaking of which,

“In June 2006, an estimated 80% of email spam was sent by zombie PCs, an increase of 30% from the prior year.”

“Zombies” being machines that have been compromised without the user’s awareness, usually by opening spam in the first place. Then the origin of spam (or kidporn, for that matter) is traced to your machine. Ouch.

“The use of botnets can be perceived as theft. The spammer consumes a zombie owner’s bandwidth and resources without any cost.”

Does this make it any more personal to you? Perhaps if your ISP identified what percentage of your bill was to cover the costs of spam and of trying to filter it… Check out the article. It isn’t perfect, but there are a lot of good citations if you want to read further.

Paeniteo • June 10, 2011 3:01 AM

@Stefan Savage: “Visa/MC are not in a position to enforce US law on foreign jurisdictions.”

OT, but they did with Wikileaks, even withour a court order (indeed, there are conflicting views on whether any US laws were violated at all – even after trying, the US did not file any charges against Assange or other Wikileaks spokespersons).
So, try again to tell me that credit card companies cannot arbitrarily shut down payments to customers…

Tim#3 • June 10, 2011 3:36 AM

It certainly seems best to go after the businesses that the spammers rely on. The only time that I have noticed a substantial decrease in spam in recent years was when the California-based hosting company McColo was taken down in 2008.

GreenSquirrel • June 10, 2011 4:18 AM

I am always intrigued by reports of spam volumes as either my ISP(s) are fantastically good at filtering or some people really are getting hammered by spam.

I have several email accounts (various work ones and multiple personal ones). None of them get much in the way of spam (*).

As an example my ISP email account hasnt delivered an item of spam to me in the last two months but has let through about 60 legtimate emails a day. When I say hasnt delivered, I mean nothing has even made it into the junk / spam folders. I am not aware of any emails being blocked as a false positive.

This means that either the ISP has a fantastic spam filtering solution with an almost miraculous rate of false positive/false negatives or, out of the last 2500 emails this account has had, not much has been spam.

I have a hotmail account that isnt as good – it sends through two or three bits of crap per day. My Yahoo! Mail account is the same but Gmail is slightly better. All three of these accounts send about 1 email into junk for every 15 or 20 I get in the inbox. All three have quite a low rate of false positives/negatives (about 1 in 50 for both types) but this is a lot higher than my ISP account.

While I am not doubting that there is a problem and a significant waste of network resources as a result of spam, the numbers always seem high to me.

This is similar to the 25% of hackers who are FBI informants (obviously a made up number given unwarranted significance and specificity) and in the Register yesterday ( http://www.theregister.co.uk/2011/06/09/cybercrime_surveys_are_tosh_says_ms/ ) MS claim that most cybercrime survey figures are nonsense.

When someone says x% of all email is spam, or “globally, spam costs US$xxxx” I am always a bit dubious.

  • here I mean “unwanted” junk mail. Where I have signed up to various companies mailing lists, I dont consider the crap they send to be spam.

Chris Kanich • June 10, 2011 9:52 AM

Impossibly Stupid,

“The vast majority of spam could actually result in a net loss; I would love to see research on that side of the fence.”

We do in fact pose exactly this hypothesis in the Spamalytics work: if spammers are ‘outsourcing’ parts of their operation at the rates we’ve seen advertised on the underground, that campaign could not have been a profitable business. The campaign we were studying however was most likely a very integrated one in which the spammer was still able to make a tidy profit.

Jorrit,

I would say that that is most likely not the case. In the course of our work we’ve been able to estimate the revenues of these spammers and they are very reasonable. In addition, it would probably be somewhat outlandish for the anti-spam companies to pose as buyers throughout the world purchasing these goods.

Paeniteo,

I think the difference in reaction there had a lot to do with incentive alignment – there were dozens of U.S. lawmakers decrying Assange and Wikileaks at the time, and at the end of the day the fees gained from a one-time outpouring of support (even in the millions) was probably not worth the hypothetical cost of being seen as aiding that organization. In addition, the government has most certainly not dropped the case against Wikileaks and Assange, as Grand Jury subpoenas are being served at an aggressive rate: http://www.salon.com/news/opinion/glenn_greenwald/2011/06/09/wikileaks

kashmarek • June 10, 2011 11:53 AM

I don’t think 100 purchases is statistically significant. They should have made more than 1000 purchases (the more the better). Otherwise, any statistical analysis can essentially be ignored.

However, since they think they have identified (some of) the banks, that means we can identify the account holder receiving the funds, and they can be shut down or held accountable (yeah, probably in your dreams).

The problem is more complex, involving marketing, suppliers, distributers, collection, and the stupid stupid buyers (in some places they too stupidly call them consumers). I think if you stop the buyers you stop the problem. Or, stop the suppliers will also do the trick. Making that work is hard. Or, you can eliminate the middlemen that make money off the deal (distributers, collectors, marketing).

Chris Kanich • June 10, 2011 12:32 PM

kashmarek,

These 100 purchases were not a random sample – they were performed to maximize the number of different programs that we purchased from. We narrowed our purchase plan down to these particular sites after extensive clustering of tens of millions of domains received in hundreds of millions of different spam messages. Tables I-IV in the paper show a good summary of the volume of different quantities involved in this filtering process.

Richard Steven Hack • June 10, 2011 2:36 PM

Green Squirrel: “MS claim that most cybercrime survey figures are nonsense.”

Uhm, and Microsoft has no vested interest in disputing the level of cybercrime since most of it depends on Windows? I think not.

I do agree that Gmail seems to be very good at dealing with spam. I have one Gmail account (the first one I opened) which I use for most Web site signups. That one is inundated with spam which Gmail catches nicely. Almost never is a legitimate email caught – and I do get some legitimate email there. A second Gmail account I use is only for signups and client contact. It gets almost no spam.

However, we have to realize that a lot of home people are still using various ISP accounts and then there are corporate email systems. So I don’t doubt the amount of spam is enormous. Whether it is 50% or 90% of all email isn’t much difference or has much relevance.

The bottom line for all this is that it is extremely hard to get rid of a criminal enterprise which is making someone sufficient money to make it worth his while. You can interdict at various points in the system, but in the end it still boils down to whether the result is worth the effort because it’s extremely unlikely you are going to stop the enterprise in general.

Again, the question is: for how much effort and cost can you reduce spam by, say, fifty percent, or 75 percent? Without some numbers on that, any intervention is questionable.

Just saying you can take out 13 banks or have the credit card companies stop dealing with spammer companies (and really, good luck with that if the credit card companies are making enough bucks – Wikileaks doe NOT compare) really means nothing if you can’t anticipate the spam networks finding a workaround.

Look at what happens when a major botnet or spam ISP IS taken down. For a short while, spam drops by a very large amount. Then guess what? It’s back again. Somehow. Anyhow.

So why should we believe that interdicting the financial end will be any more effective than interdicting the botnet or ISP? Why can we assume that can’t be worked around? I don’t see any proof here that this will work any more than it works with Iran sanctions.

And you can’t know if they can find a workaround until you actually do it.

A Google search found this:

How much do spammers actually make?
http://blogs.msdn.com/b/tzink/archive/2008/08/28/how-much-do-spammers-actually-make.aspx

That guy made probably over $300K per year. Another study reported by Wired – http://www.wired.com/magazine/2011/02/st_equation_spamprofits/ – suggested $7,000 a DAY.

Multiply that by however many spammers there are who are just as good – a number we can’t know. It’s still going to be a big number. Big enough that those making that kind of money are going to find ways around any deterrents.

Look at the efforts spammers make to evade anti-spam software. They’ll take just as intense efforts to evade being cut off from the financial system.

I’m not saying it shouldn’t be tried. I’m saying I’d like to see a more detailed plan with real numbers – and some effort in the plan to deal with the inevitable evasion attempts.

David Emery • June 10, 2011 2:59 PM

Fascinating stuff! Do you think the consolidation of payments through a few banks means that (a) the back end operations are similarly consolidated or (b) these are the banks that the ‘wide underground’ knows are safe to use?

Stefan Savage • June 10, 2011 4:30 PM

David,
Back-end operations are consolidated within sponsoring affiliate programs (e.g., all zed-cash brand sites, regardless of the affiliate, are cleared through the same back-end, although they may use different banks depending on the type of goods). There is also some sharing of back-end capacity among different programs with joint ownership (e.g., rx-partners/stimul-cash/mailien). Finally, its pretty clear that there are some banks/processors who are understood to be more receptive to certain kinds of high-risk product categories and are being shared independent of any operational sharing between programs.

Grumpy • June 10, 2011 5:39 PM

@Green Squirrel

I am administrating a number of E-mail servers in different locations and see a good amount of spam.

As an example: One system (a primary mail server and a secondary/backup mailserver with lower delivery priority in the MX record) with a few domains (two carrying most of the traffic and four or five with little or no traffic) receive on average more than 10000 spam mails a month per server, compared to a traffic of about 2000 legitimate mails a month, i.e. more than 90% of all incoming mail is spam!

Each spam mail has to be received by the spam filter (i.e. consumes bandwidth) it is filtered (consumes CPU and other resources) and sometimes has to be searched for (the extremely rare) cases of false positives (consumes manpower).

That, to me, is quite tangible proof that spam figures are probably not overly exaggerated. The spam to non-spam ratio is presumably higher here than for systems with higher volumes of legitimate mail, but at other servers I rarely see less than 70+% spam!

GreenSquirrel • June 10, 2011 6:06 PM

@RSH / @Grumpy

I never meant to say I fully backed the MS claims, but I do think there is a massive (astronomically large) amount of FUD when it comes to “cyber”-crime related figures.

The point I was making about my email accounts is that I am not even seeing mail go into the spam folders. So if my ISP account, Hotmail account, Yahoo account or gmail account are getting even 50% spam, then this is being deleted away from anywhere I can intervene. However, if this is going on, I am more than a little concerned that there must be legitimate mail getting deleted as well. Unless of course the filters are so accurate they have a zero false positive rate.

In total, including news letters I have signed up to, job alerts, and marketing from companies I have purchased things from (and for whatever crazy reason agreed to their mailing lists), I get in the region of 150 emails to my personal accounts each day.

Using the last weeks figures (all I have), there are a total of 28 across all four accounts. All of this spam has ended up in the junk mail folders as has 1 legitimate email from a recruitment agency.

So in the last 7 days, I have had about 1100 emails, of which 30 have been spam.

Should I assume then, that either I am absurdly lucky when it comes to spam or the spam is being killed before it hits my account. If it is being killed early (and given that the spam I get is false positive about 1 in 30 times) how do I go about finding out where the missing false positives are?

Brian • June 10, 2011 7:21 PM

This discussion reminds me of the form responses you see on slashdot when spam discussions come up.

“It appears that you are suggesting a
[ ] foo
[ ] bar
…
response to spam.”

etc…

Richard Steven Hack • June 11, 2011 2:44 PM

Green Squirrel: Re losing mail to external spam filters, I had a client once who used one of these external email hosters who claimed their spam filters were “100% effective.” The client claimed there were losing legitimate email. They had no recent examples and I couldn’t see any problem with their settings on the external host.

The external host claimed your email NEVER resided on any of their servers – it was all processed through multiple spam and AV filters in memory apparently through a chain of servers.

It was my opinion then that there was no way they could do that without losing some legitimate email. I advised the client to get in touch with their email hoster (as well as the recipients of the “lost” email) and have them verify that a given “lost” email transited all the servers between the client and the recipient. This is the only way to determine if an email has actually been “lost” and how.

However, these days seeing how well Gmail handles spam, I have to say it seems possible that spam filters are REALLY REALLY good, at least at Gmail’s end.

As for the amount of spam hitting my spam mailbox, it is VASTLY greater than the legitimate email. But again, that’s because I don’t use that mailbox for anything but spam. My other account is only used for services that are unlikely to sell or allow my email addy to be harvested. I think that’s the primary difference.

That and the fact that I do VERY little email in general than most people do – I have no friends and little business…:-)

I think the amount of spam one gets is directly proportional to how networked one is. A company by definition is far more networked than any individual, so they get far more spam than an individual. A highly networked individual gets a lot more spam than I do.

I get maybe a dozen or two emails a week in my “legitimate” account, of which almost none are spam – and almost no spam in that Gmail spam trap either. I don’t count the SANS announcements I’m too lazy to unsubscribe from although they appear in both accounts.

In my Gmail spam account, I get maybe fifty emails a week, many of which are Twitter notifications of new followers, plus various emails from Web sites I signed up for no really good reason.

Right this instant I have 35 emails in the spam account that are “not spam” – most are from Computerworld Resources, Infoworld Resources, Twitter, Yelp, Groupon, IDG Connect, The SANS Institute, etc. There is about four pieces I would consider spam since I never signed up for them AFAIK.

In the spam folder right now are 72 pieces of spam – almost exactly double the legit email by coincidence.

I clean out those folders every time I go into that account, but I can say that the amount of spam will go up faster than the amount of “legitimate” emails in that account.

The other account is so far today completely empty in both legitimate and spam.

I just cleaned out the spam account – and took advantage of Google’s offer to unsubscribe me from some of those email news subscriptions, which I didn’t know it could do. Now I should have even less “legitimate” email in that account.

Grumpy • June 11, 2011 6:11 PM

@Green Squirrel

I just realised that we have been comparing apples and oranges here.

I was talking about spam on server level (actually more like domain level) whereas you were talking about spam on account level.

Most of the spam received is actually to non-existing accounts — the remainder of spam sent to actual mailboxes is far lower. I have no recent figure, but around 10 – 30 a day is probably not far off the mark.

From an analysis I did last year, using one-time addresses, I found that small traders often were involved, presumably due to vira and/or trojans. Amazon’s marketplace (or whatever it is called) seemed to be especially bad, much worse than e.g. eBay.

And obviously E-mail addresses published on the Internet are prone to spam. (I made a silly mistake some time back and used my main E-mail address when submitting changes to an open source project and the maintainer gave me (in this case unwanted) attribution in the source code, which was (naturally!) exposed on Internet. Spam-ma-bang!)

It was far from a “scientific” analysis, but it did help me get a handle on many of the causes of spam.

Doug Coulter • June 11, 2011 8:50 PM

Big mail servers have one huge advantage over any filter I could run here — lots and lots of customers.

This adds a simple metric — did this incoming mail go to a lot of the customers, and as above, a lot to null accounts? Surely spam then.

In other words, the better they are at spamming everyone, the easier they are to handle.

Oisín • June 16, 2011 7:35 PM

@kashmarek
“I don’t think 100 purchases is statistically significant. They should have made more than 1000 purchases (the more the better). Otherwise, any statistical analysis can essentially be ignored.”

Please don’t make criticisms based on subjects you clearly know little about, because it makes it less likely that people will consider or even read your other arguments, even if they are valid.

It makes no sense to say “oh 100 samples is statistically insignificant, let’s go with 1000. Yeah, that should be good”.
When statistical analysis is used to draw conclusions from experiments, there are well-defined mathematical tools we use to ask “if our assumption is wrong or unfounded, then what is the likelihood of arriving at the measured values (given the sample size, the observed variance between and within samples and sample groups, etc)”.

There is no universal “significant sample size”.

Andy • June 16, 2011 8:40 PM

“95% of the 76” 7676 = 577695% = 5487 * 76 = 417012 unique elements
or 5487 unique elements per order

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.