Tracing Spam from E-mail Headers

Interesting article from Brian Krebs.

Posted on April 21, 2017 at 6:22 AM • 30 Comments


hhsecwatchApril 21, 2017 7:15 AM

Given the nature of the affected org here, I would almost be concerned that this is a smokescreen. While their incident response teams are running around dealing with PR etc. something more serious could still be going on.

In fact I’d bet five bucks on it.

Patriot COMSECApril 21, 2017 7:18 AM

It is pretty funny, but it is also pathetic. I like the aptly named "Trace Systems"--what a hoot!

So, you sell U.S. Government-related security products? What good are you? You can send my spam.

rApril 21, 2017 7:35 AM


"they were inside of a U.S. defense contractor which specializes in 'custom military-grade communications'"

My sentiments exactly

Custom lol, they obviously haven't been briefed on the ins and outs of the CIA papers.

Clive RobinsonApril 21, 2017 8:41 AM

I'm not surprised this has happened.

Few people if anybody likes doing "mail server duty" especially in large organisations as keeping an eye on logs and other indicators takes time out from other activities. Some use automation but due to false positives, have a habit of 'turning the gain down' untill they don't get disturbed, by things they do not want to see or in many cases do not understand in sufficient detail.

Further the use of Firewall rules to keep out the unwanted in theory is simple... That is untill the director of Marketing want's his "road warriors" to "send on the go" through the company gateway or some such.

Thus those with a compromised mail server may often be the last to know...

Slime Mold with MustardApril 21, 2017 4:16 PM

I read the article and the comments, but never learned How Movie & Pop Stars Are Still In Shape ; >

Concern that login credentials were compromised are, I hope, outdated. Slowly, the offices around me have adopted our model of separating internet connected and 'mission critical' (intranet) machines. Lots of procedures, but better security would probably involve killing off the staff.

Ron GApril 21, 2017 4:51 PM

I just thought that I should clarify a couple of important points about the Krebs story. (And yes, I am the guy who brought it to his attention.)

Firstly, this isn't a "smokescreen" for something more nefarious. I had already been tracking these exact spammers for some time, and they routinely manage to abscond with login credentials of random folks and then use those to send spams out of the (un-blacklisted) mail servers of various/numerous "legitimate" companies. This has been going on for some time now.

Second, focusing on the tedious life of a typical mail server admin in this case misses the point entirely. This guy dan(at) undoubtely clicked on a link (which he shouldn't have done) in a spam email that he'd recceived, and a few milliseconds later he got a fistfull of exploit JavaScript back and from that point on, he and his personal workstation were, as they say, totally pwned. (This is a pattern of behaviour I have already found to be associated with this particular spammer, i.e. sending down JavaScript exploits to anybody dumb enough to click on their links.) The wisdom, or lack thereof, of allowing your typical ignorant/stupid employees to read their emails on Windoze systems, particularly if your company is claiming to have high level cybersecurity expertise is a rant that I'll have to complete on some other day. For today however I'd just like to remind everyone that at least one study claims to have shown that even users who have been educated on the risks of clicking on links in spam emails still do so a large percentage of the time. One can only hope that someday, these kinds of megabuck "beltway bandit" companies that charge the taxpayer up the wazoo for their gold-palted "cybersecurity expertise" will take heed and stop allowing their un-watched, un-tested, and possibly also un-educated employees to read their emails on un-patched Windoze systems.

So anyway, Mr. Dan was totally pwned, whereupon the spammer, via an undoubtedly automated process, rummaged around on Dan's harddrive in search of mail server authentication credentials, and apparently found a set of those and then transmitted those back to the mothership, after which the spammer used them in exactly the same way as I've already seen him use many others, specifically to send out yet more spam with yet more links leading to yet more exploits.

But this is *not* just a mail server issue or an issue that should only be of concern to the company's mail server admin. If you look at the web site associated with you'll see hat it is in fact a "portal" system which is undoubtedly used by the various DoD contractors and subcontractors who are working on the DoD's next-gen *battlefield* communications system. These contractors and subcontractors, which include Northrop Grumman, Raytheon, Lockeed Martin, General Dynamics, Harris and a raft of others obviously set up this "potral" web site as a facility that would let them all log in and communicate with one another about the project.

Google for "GTACS contract"... and be prepared to be horrified when you realize that the set of credentials that the inept Mr. Dan @ allowed a lowly spammer to make off were in fact, in all probability, also a set of -login- credentials which could have been used to get in and, at the very least, hobnob with all of the technical folks who are working on the DoD's next-gen battlefile communications system. At worst, of course, those same credentials might possibly have allowed access to plans, designs, schematics, encryption keys (for early-stage GTACS prototypes) and so forth.

And as if all that wasn't horrifying enough, please note also that it is self-evedent from the known facts in this case that are clearly *not* using something like a two-factor identification system, e.g. ones involving a personal crypto keyfob. Add to this list of boneheaded security faux pas also the fact that the company apparently allows people to log in and use/abuse their servers, apparently without restriction from anywhere in the world. (The Russian spammer in this case exploited the mail server from an anonymous IPv4 address located in Brazil.)

Despite all of the above travesties, I have not yet even mentioned the real and ultimate take-away of this story. That is just this: If a lowly spammer seeking only to send out more spam (without it being blocked) could, almost by accident, so easily get this far in exploiting a DoD contractor company... and one that's building the DoD's next-gen battelfield communication system... then how far could a really *determined* adversary get?
It boggles the mind to think about it.

And it isn't as if these defense contractor companies have an exactly sterling reputation for maintaining tight security on their networks. Far from it. The problem got so bad, apparently, that congress saw fit to pass a law a few years ago which would force the companies to at least -report- these kinds of security incidents... which by and large they were apparently loath to do.

Clive RobinsonApril 21, 2017 6:07 PM

@ Slime Mould...,

... but better security would probably involve killing off the staff.

You don't need to be that drastic.

Just follow the forward thinking policy of the TSA and airlines towards passengers,

When they arrive, que them up totaly humiliate them by getting them to strip off compleatly and put on those back opening hospital gowns, grope them up, then give them full cavity searches. After that bit of fun move onto the scary bit, strap then to a light weight frame stretcher, drug them and then stack them like cord wood in containers and load them like dead weight freight. Send somebody around with one of those spray gun inoculators and if somebody starts to moan / scream / jibber give them a quick shot to put them out again.

When you arive at the end you give them a shot of the antidote to wake them up, then hurd them naked into a communal toilet/ shower then let them dress and go on their way...

Nice and simple and they are no problem, as they don't need to be fed etc, nor move so you don't need a galley or stewards or that space wasting furniture so you can pack more people in.

In fact almost as many as you get on the London Underground Northern Line during rush hour, where if you are short having your nose pushed in a sweaty armpit is par for the course. Or if you are tall as I am, having the hair sucked off your head by the air circulator you smash your head into as you are forced down the carriage at each successive station by more people getting on...

JJApril 21, 2017 6:30 PM

Ron G

>ending down JavaScript exploits to anybody dumb enough to click on their links.) >The wisdom, or lack thereof, of allowing your typical ignorant/stupid employees >to read their emails on Windoze systems,

it's even more simple. One doesn't need 'Schneier Hall of Fame' status to defeat this. But a simple FOSS browser add-on like Ublock Origin, or a malware/anti virus blocking dodgy websites. The latter have many issues but for a reduced threat model they drastically reduce the relevant attack surface. Why on earth
is not configured properly, as you say, is the smoking gun

Of course the rudimentary practice of reading the characters in the link is the prime lesson for everyone

rApril 21, 2017 8:20 PM


That doesn't stop non script actors, bottom line: don't click links.

IMG'n that...

Impossibly StupidApril 21, 2017 8:59 PM

I used to post the same sort of detective work on my blog for both spam directed towards its commenting system (or other exploit attempts that got logged) as well as regular email spam/abuse. But there are just too many insecure systems in the world that nobody gives a damn about fixing, so I now mainly just drop the offenders into the firewall and move on. Until companies start offering cash rewards for abuse reports, I really can't be bothered to do their "security" for them.

JPAApril 21, 2017 9:54 PM

I'm a little confused by Ron G's statement that the person got a " fistfull of exploit JavaScript back and from that point on, he and his personal workstation were, as they say, totally pwned"

I was under the impression that while Java could get access to install malware JavaScript did not have that deep access to the OS. While JS can be used for things like cross-site scripting and obtain credentials that way it should not be able to take over a computer or workstation.

I would be grateful for some clarification.


ab praeceptisApril 22, 2017 4:03 AM

Ron G

Well, it's known since many, many years that email is a very poor communications means security-wise.

Some time ago it became en vogue to adhere to the sectarian belief that adding ssl/tls to it would somehow make it more secure. Ok, there are facts disturbing that belief somewhat but since when do facts stop the ssl believers?

Yet, I was discontent as there was still some remains of reason stubbornly standing in the way, just like a convinced scientist shouting out in front of a temple "It's not angels but physical process that move our cars!" - and then came webmail and web 2.0. Now I'm content. Finally we are well protected and can fully enjoy our webmail experience!

I can find lustful and beautiful women, too! They are just a click away! As soon as all those lustful beautiful young ladies besiege me there would be a problem, but - note the "would" - just like the women viagra is also just a click away in webmail happiness (TM).
And the flow happiness doesn't end there. I'll also be very rich very soon as I have been informed to be the one in a million in not just one lottery! And all I had to do was to click.

Of course I know that there are still dangers out there. I'm not stupid! But I don't worry as I have ordered and installed the symerski security experience package. So, let them come, those hackers and whatnot; they won't succeed to break my golden protector seal!

The paradise exists - and we are already in it with one foot!

ab praeceptisApril 22, 2017 4:25 AM

“If he did realize what he has in his hands, I feel sure that the Russians and/or the Chinese would be more than happy to buy those credentials from him, probably reimbursing him more for those than any amount he could hope to make in years of spamming.”

krebs is a security GURU, not a mere "security expert"! as the above quote amply demonstrates!

OK, just a couple of lines earlier he described the "hackzors" as inept dumbheads (my wording) but a security can't be bothered with mere mortal criteria like basic logic.

Usually I don't read krebs; I'm not interested in "Batman for idiots" kind of "security news". But as our host linked to it I had a look - and I was not disappointed. Lots of really good hardcore security pron there.

Like "...usually unseen addressing and routing details that accompany each message. They’re generally unseen because they’re hidden unless you know how and where to look for them."
Darn! Hard times for cyber-crooks! You don't fool krebs, that's for sure. As the link he provides shows, his Batman apprentice toolset is stuffed! Just clicking and spotting your email client, say windows outlook xyz, one, or in that case, krebs holds the master recipee in his hands and *bang* the "usually unseen addressing and routing details" are in plain sight to the master guru!

But I have to stop, I'm kind of shell shocked by the image from what seems to be trace systems invincible-cyber-security-center. Just look at that guy in the center. Like a fighter jet pilot, he wears a head set incl. microphone! I guess that's for hardcore scenarios where the cyber-teamleader coordinates traces cyber-team to press Ctrl-H to look at the "usually unseen addressing and routing details".

Darn. The russian and chinese state hackers can go pack. No chance against that kind of cyber-batman-security - with pilot like headset!.

I'd LOVE to have a protective golden sticker from them but I guess that kind of sticker is way above my paygrade and only given to the really, really, big security gurus and then only for tens of millions of $ and an nda by webmail 2.0. Damnit.

WinterApril 22, 2017 5:59 AM

@ab praeceptis
"Usually I don't read krebs; I'm not interested in "Batman for idiots" kind of "security news". "

Krebs is a journalist who writes not for you, but for mere mortals. Which means he has to select his examples to be comprehensible for those who are unable to follow the "real" security news like this blog.

And if I had to select groups as examples of those willing and able to pay good money for login credentials in the Beltway, I too would go for Russians and Chinese. Those are good examples for the mere mortals who read his articles.

Also, Krebs never portraits himself as a security expert, but as an investigating journalist. And I think he is good at it. He certainly is a much, much better journalist than I am, and I guess also better than you are.

Everyone his own trade.

Clive RobinsonApril 22, 2017 7:36 AM

@ ab praeceptis,

I'm at the London Science Museum today with the not so little "little one".

They have a little side exhibition about data for "mear mortals" and their off spring. They've revamped it since last I saw it and it has all sorts of helpfulness such as,

    Your 'likes' are data and they're giving away more than you think

But as with all such advice it's more a platitude than an action point.

Perhaps you could send them one of your golden stickers :-D

Dirk PraetApril 22, 2017 8:12 AM

@ Winter

Also, Krebs never portraits himself as a security expert, but as an investigating journalist. And I think he is good at it.

Same feeling here. The man does really fine investigative work, and writes about it in a way mere mortals, and even LEOs and politicians can understand. Thus very much contributing to creating security awareness with a broad audience. We need more guys like him.

ab praeceptisApril 22, 2017 8:36 AM

Winter, Dirk Praet and probably others

My joking was mainly about those utterly ridiculous "cyber security" corps close to the us of a doiw (department for illegal wars) and spook agencies.

As for krebs, I absolutely see that there is interest for that kind of gazette. That's not the point.

I do, however, expect a reasonable minimum level of consistency and logic. To paint, for instance, some hackzors as ridiculous imbeciles but then, some sentences further on, to assert that the russian and chinese agencies would pay big money for what the imbecile achieved is a) showing krebs' article being imbecile and b) insulting russian and chinese agencies.

ab praeceptisApril 22, 2017 8:54 AM

Clive Robinson

I'd love to help out with a silver "science for everyone" sticker but I'm a lousy and poor designer.

Seriously though: I *welcome* endeavours to explain science and technology to the (uneducated) masses. Moreover I have a rather high tolerance level for low level gazettes. They have their readership and those are citizens, too; may they enjoy their read (or gaze).

krebs, however, is something different. And I'm beginning to actively dislike that guy because of his tendency to paint fbi and other agencies as the good ones and the Russians and Chinese as the eternally evil ones who would just love to eavesdrop on and steal the oh so valuable knowledge of the us-americans if only they weren't so utterly stupid and left with stealing and mindlessly hacking only.

Most us-americans might not see or ignore it, but endless series of insults and baseless accusations and of dirty propaganda (and, in fact, war-like sanctions) against powers who are, in fact, in no way inferior (rather the contrary) might not go unpunished forever. Even more so as the us-americans love to threaten - incl. mil actions.

Whatever. krebs is free to write what he pleases and I'm free to show how idiotic some of his utterings are.

WinterApril 22, 2017 9:16 AM

@ab praeceptis
Krebs is a US American. The FBI&CIA are his team and they keep him safe, to some extend.

Countries like the USA, Russia, and China (and every other country) have conflicting interests that lead to spying and even aggression at times. The TLAs from the USA are to a certain extend the potential enemies of all non USA citizens, just as the corresponding institutions in Russia, China, etc. are the potential enemies of every other human.

Which boils down to the simple conclusion that a hacker that stumbles over the login credentials of US cybersecurity systems could fetch a good price from spy agencies like from Russia and China (and many others). Not seeing this makes this hacker utterly incompetent.

I do not see any problem with Krebs portraying Russia as an "enemy" state towards the USA. There is ample evidence that Russian spies have actually intervened violently in foreign countries, the US included. Russia and China make the same claims against the USA, and no halfway reasonable mind will deny these things happen on all sides.

But we are all partizan about which actions we feel are to our benefits and detriment.

ab praeceptisApril 22, 2017 10:37 AM


If some third rate hacker can get at those credentials than Russia, China, and Zimbabwe can get at them, too. Sorry if that shocks you but from what I see, besides lots and lots of noise, hollywood crap, and "cyber guards" with pilot headsets (plus a ridiculous setting), the reality seems to be that the security of critical us systems is pretty much lousy to non existant.

Moreover, while I do not think that the russian and chinese intelligence agencies do not at all hack the us of a, it seems noteworthy that *factually* quite little is known about their attacks, while the us of a are very well known to hack everybody incl. their "allies". And sorry, but noise and payed liars, err, the "media" is *not* proof.

As for aggression: Tell me about illegal wars Russia or China started. Where is that list? On the other hand the list of atrocities and illegal wars and mass-murders committed by the us of a is quite long and stretches pretty much around the globe.

Let's keep it simple. Let's stick to credible evidence and to basic logic. krebs doesn't meet that as I have demonstrated. Some of what he writes may be good or at least acceptable but some of it is plain bullsh*t and illogical.

WinterApril 22, 2017 11:14 AM

"Tell me about illegal wars Russia or China started."

Murdering opposition activists in London, the Crimea (those green men are not Russian soldiers), Georgia, several times, Ukraine (we did not shoot down MH17, and besides, that plane was filled with corpses in Amsterdam). Helpng Assad with chemical attacks.

And that was just Russia.

Do not try to fool us. We are not limited to Russia Today for our news, nor US media.

ab praeceptisApril 22, 2017 11:50 AM


For reasons of respect for our host and the rules here, my response can be found in the squid thread (so as to keep matters unrelated to the topic here outside).

Clive RobinsonApril 22, 2017 1:13 PM

@ Winter,

As far as selling the details to the agent of a foreign power, I realy do not think Brian Krebs has thought it through.

If the hacker sticks to just spaming they are basicaly little more than a nuisance. Thus they will not realy be a matter of urgency or concern to either US LEO or US IC entities.

If however they sold them to the agents of a foreign power then they would be hanging from two hooks. The first is the agents of a foreign power would be able to blackmail them for ever (just as the FBI tend to do with hackers). Secondly they make it to various "Top Ten Most Wanted Lists" for many US LEO and IC entities.

This would be especialy true if the account belongs to a person who has ever worked for the IC as revealing their identity is "treason" for which the US still reserves capital punishment.

Even if not, the account holder is more than likely in a sensitive position, which is just as bad. To see why look at the current hoops and loops the show dogs of the various US entities are jumping through to have the "public face" of Wikileaks charged with treason.

Thus I think Brian Krebs is not being a suitably aware, let alone a good journalist or security reporter on this point.

JPAApril 22, 2017 4:27 PM


Thanks very much for posting the link on the stack overflow exploit. Its educational reading for me.

RatioApril 22, 2017 10:29 PM

@ab praeceptis,

“If he did realize what he has in his hands, I feel sure that the Russians and/or the Chinese would be more than happy to buy those credentials from him, probably reimbursing him more for those than any amount he could hope to make in years of spamming.”

krebs is a security GURU, not a mere "security expert"! as the above quote amply demonstrates!

So Brian Krebs quoting Ron Guilmette makes him a security GURU.

You then quote Krebs's quotation, presumably making you a security GURU.

I'm not sure, but I kinda feel like I just moved closer to being a security GURU just by quoting your quotation of Krebs's quotation of Guilmette. Powerful stuff, this quoting.

@Clive Robinson,

If the hacker sticks to just spaming they are basicaly little more than a nuisance. Thus they will not realy be a matter of urgency or concern to either US LEO or US IC entities. If however they sold them to the agents of a foreign power then they would be hanging from two hooks.

Only if they can get their hands on them. Otherwise, not so much.

ab praeceptisApril 23, 2017 4:15 AM


Bad news. Nope, you and I are not security gurus because we don't know Ron G. Moreover we (I take the liberty to presume for you, too) do not wear pilot head sets in our situation room (do we have one?) when tracing expertly uncovered (Ctrl-H) email headers.

But there is at least one thing that I have (presumably you, too) that may serve as an excuse to consider me (us) as at least remotely security related: I (we, I presume) actually understand what Ron G. wrote. So, maybe we could be journo GURUS.

Should I get us golden stickers printed?

A. NonApril 23, 2017 8:40 AM

The article you cite is written by an idiot. Reply-To: is one of the easiest headers to forge. Spammers will often pick a valid email address from their dictionary and stick it in the Reply-To field. The only field you can reasonably count on is the top-most Received field, which is written by your own email server. If you doubt this, let me know and I will be glad to send you a forged email with whatever you stipulate in the Reply-To: field.

Peter A.April 24, 2017 8:44 AM

@A. Non

I am unable to check DKIM-Signature in the published headers (as Brian K. has modified them) but I hope Ron G. did. If it is valid, it means the mail has traversed the mail servers responsible for domain's mail which have control of the domain's signing key, specifically the mail submission service, because it's the source that get the emails DKIM-signed. This means valid credentials were used to log in to the submission service. Likely, it also means that Mr. Dan's credentials were used to log in, but it depends on the servers configuration - if it enforces From: header to match login credentials. If it doesn't, the spammer could have potentially used some other credentials but faked the From: header - but it is just as problematic.

Ron GApril 24, 2017 2:27 PM

My thanks to Ratio for pointing out that it was actually -me- who made the comment (to Brian Krebs) that, in my opinion, if the spammer had realized that he had managed to make off with a set of credentials for this particular DoD / Homeland Security contractor company, then he almost certainly could have gotten a pretty penny in exchange for those if he had shopped them around. And to my way of thinking, the Russians and the Chinese intelligence services would, in all probability, have been likely bidders in any such hypothetical auction, just as U.S. intelligence and Russian intelligence would be likely bidders for sets of credentials useable to get into Chinese PLA systems.

My comment was not meant to pass any sort of judgement, moral or otherwise, on any party. The intelligence game is what it is, worldwide, and everybody plays it.

In reply to A. Non, let me just say that actually, in a case like this, the topmost -two- Received: headers (and also all other headers -above- that second Received: header) are believable / trustworthy for the simple reason that we have no reason to believe that the main outbound mail server has been totally compromised -or- that is itself the kind of company that would have, or that does have any motivation for forging headers... unlike numerous spammers I know.

The topmost Received: header was pre-pended onto the spam message by my own mail server. I trust me not to lie to me, and I likewise trust my own mail server not to lie to me. So from my perspective, at least, that topmost Received: header is very nearly 100% trustworthy. The second Received: header was pre-pended onto the message by mail server, and as I have just noted, there is no reason... clear or otherwise... at this point to think that or its mail server had any motivation to fiddle that or to put bogus info into that header. So we can categorize that one also as "most likely trustworthy".

Of course, *in theory*, I could have entirely made this whole thing up, and I could have written every bit of the message headers shown in the Krebs article in an ordinary text editor. In theory, it could all just be a big hoax and I could just be having everybody on, including even Brian. But in that case, I think that the PR department at Trace Systems would have been quick to denounce me as a hoaxer. But Brian independently communicated with the company, and it would appear that at no time did they dispute that such a mail message had traveled through their system, or that any such user as dan(at) existed, or that he had some credentials to use their mail system, or that those credentials had in fact been used to perform TLS authentication (as shown in the second Received: header) on or about April 12, 2017. Instead, the essence of the company's comments on this incident seem to be that the account in question has now been disabled.

In response to Peter A., no I didn't do any checking on DKIM in this case. There's didn't seem to be any need to do for the reasons I've just explained. From where I am sitting, the topmost -two- Received: headers appear to be entirely genuine and believable.

{{WARNING: Rant ahead}}

P.S. The pictures of the dedicated soldiers and sailors in the images that Brian swiped from the relevant web sites just reminded me of something. Those are probably *real* soldiers and sailors... honorable men & women willing to risk life and limb to keep us all safe... to keep *me* safe. But -their- safety relies on the people back home, like the people at Trace Systems, who one hopes will do their jobs, with meticulous care, and *not* with their first thoughts every day being about how they can shave a few extra pennies off their security budget so that they can instead put it into their own pockets.

Well, that's what one hopes anyway. But to quote from my favorite book "Everybody works for Milo."

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of IBM Resilient.