Another Massive Russian Hack of US Government Networks

The press is reporting a massive hack of US government networks by sophisticated Russian hackers.

Officials said a hunt was on to determine if other parts of the government had been affected by what looked to be one of the most sophisticated, and perhaps among the largest, attacks on federal systems in the past five years. Several said national security-related agencies were also targeted, though it was not clear whether the systems contained highly classified material.


The motive for the attack on the agency and the Treasury Department remains elusive, two people familiar with the matter said. One government official said it was too soon to tell how damaging the attacks were and how much material was lost, but according to several corporate officials, the attacks had been underway as early as this spring, meaning they continued undetected through months of the pandemic and the election season.

The attack vector seems to be a malicious update in SolarWinds’ “Orion” IT monitoring platform, which is widely used in the US government (and elsewhere).

SolarWinds’ comprehensive products and services are used by more than 300,000 customers worldwide, including military, Fortune 500 companies, government agencies, and education institutions. Our customer list includes:

  • More than 425 of the US Fortune 500
  • All ten of the top ten US telecommunications companies
  • All five branches of the US Military
  • The US Pentagon, State Department, NASA, NSA, Postal Service, NOAA, Department of Justice, and the Office of the President of the United States
  • All five of the top five US accounting firms
  • Hundreds of universities and colleges worldwide

I’m sure more details will become public over the next several weeks.

EDITED TO ADD (12/15): More news.

Posted on December 15, 2020 at 6:44 AM45 Comments


Sancho_P December 15, 2020 7:31 AM

Since years,
and they still can’t defend themselves against sophisticated Russian hackers?
The founders of the Net?

wiredog December 15, 2020 8:33 AM

What’s really interesting is that no one is officially saying anything, and very few are saying anything unofficially. Even on the various high sides no one is saying anything.

This is bad. Seriously. The US (and others, because if you think that only the US was hit then I’ve got a bridge to sell you) will probably be forced to respond strongly.

I wonder how Russia would cope with the sort of economic sanctions Iran has been dealing with?

BP December 15, 2020 8:38 AM

What the heck is Silverado Policy Accelerator. Supposedly a think tank as described on pbs news. How you you have ideology and security in the same unit. Are you security guys infected with the money men who like to fund think tanks and make truth disappear?

David Rudling December 15, 2020 8:59 AM

This entirely justifies the comment by @Clive Robinson about the correct priorities of a cybersecurity policy agenda last week.

ht tps://

Tom December 15, 2020 9:14 AM

I work for a defense contractor. First question I was asking was how a hacker could get malicious code into a signed software update package. Answer I received was that a Russian agent was unknowing employed by Solarwinds and that is where the ability came from. Insider threats are the hardest to protect against.

Clive Robinson December 15, 2020 10:52 AM

@ ALL,

First of can we stop calling it a “supply chain attack” because that can mean anything much like “Going on a trip” does.

What has happend at SolarWind is no less than one of the most serious things that can happen to a system based on cryptography (which code signing is). That is it is a successful Keying Material (KeyMat) attack which due to the nature of PK Certs used for signing is worse than that, as it’s also a successful KeyMat Generation (KeyGen) and KeyMat Managment (KeyMan) attack.

The purpose of which was to put a “covert backdoor” into around 1/3rd of a million Internet connrcted sites World Wide, not just in the US. With about 2/3rds effected, not just Government and Millitary sites but many major commercial sites. Thus the value of this attack is quite literally incalculable…

So the probability is not that it was a “rouge insider” attack, but something managment were aware of, as unless SolarWind is compleately screwed up in the development, test, tech support, and maintainence areas this should have raised all sorts of questions, that only a senior manager could keep under wraps.

So a question to ponder as this has “Global Scope” not just “US Scope” and would have needed the buy in of atleast one Senior Manager very much against the companies interests, who could do that…

The answer most likely on the technical side is, a US based entity possibly an IC or Federal LEO entity. We know this happens RSA, Jupiter Networks and others have been caught out. US based Federal entities can do it with little more than a National Security letter co-opting the SolarWind staff (something the DoJ/FBI tried with Apple, that ended up in a nasty court case the DoJ/FBI would have lost and got adverse case law, thus they bailed out on).

Now think what’s required for even a covert insider to do… Then an outside entity… Yup you will realise that the simplest technical explanation is it was an inside process from a senior manager in SolarWind. Which leaves the human question of “Why?” or “On who’s behalf?”.

Can we narrow it down, well peobably even though there is very little to go on. For all people are saying this is not realy a sophisticated attack of the sort we have seen with an IC Entity. That is they cover their tracks way better than Federal LEO entities have ever done.

So I’m guessing that whilst we might see silly indictments as political stunts, which might de a good way for the FBI/DoJ to cover it up. The real story if we ever get to hear it will be way more interesting.

Thus it might be evidence of a failed “LEO Backdoor” where the information about the backdoor “leaked out” and got exploited by others much to the potential embarrassment of those who keep telling people to “neerd harder” rather than accept the reality of a backdoor is it alows both ingress and egress from who ever knows about it…

Before anyone asks, this is a hypothesis based on scant information and the application of William of Ockham’s notion[1], we now call Occham’s Razor. At the moment the little we know about the technical asspects of this attack are that those involved appear to be trying to keep them obscure or covered up for reasons currently unknown.

[1] Alledgedly he was born a few miles doen the road from me a few centuries ago when lifes choices for those not born into the land owning classes, was labour on the land, soldier craftmanship, priest. With life expectancy being about in that order as well.

Clive Robinson December 15, 2020 11:08 AM


On reading another link I realise that my above saying 1/3rd should say 1/30th (the article I read from another source said 180k for the potentially affected not 18k other articles say.

Impossibly Stupid December 15, 2020 12:00 PM

@Clive Robinson

So the probability is not that it was a “rouge insider” attack, but something managment were aware of, as unless SolarWind is compleately screwed up in the development, test, tech support, and maintainence areas this should have raised all sorts of questions, that only a senior manager could keep under wraps.

There is a wealth of management incompetence in a lot of US companies, and that usually extends in spades to the hiring of technical staff, assuming they just don’t outsource much of their development work.

Thus it might be evidence of a failed “LEO Backdoor”

I’d agree that’s a reasonable speculation. It could also highlight just how toxic the philosophy of backdoors is. That is to say, once there is an acceptance of the culture that allows backdoors, people who might otherwise be security conscious might just ignore the one placed by a “rouge insider” if it’s just another one that got added to a pile of approved backdoors.

Etienne December 15, 2020 12:32 PM

I believe a response is required from the USA, if not worldwide.

That response being the removal of all Russian ASN routes and their Proxies.

If you are not civilized, you can’t participate.

humdee December 15, 2020 3:34 PM


Cloak and dagger
Dagger and cloak
Attribution who?
It’s an internet joke.

Spy vs spy
Lie vs lie
The truth is who?
Wink your eye

It’s all pointers in a database
Tracing symbols in a futile chase.
A windup dancer in a pirouette
Not Russian, attribution roulette.

SpaceLifeForm December 15, 2020 8:48 PM

Microsoft now has sinkholed the C2 domain.

And tomorrow, they are rolling out an update to Defender that will quarantine the executables from SolarWinds.

So, it just a matter of time for orgs to discover that they are using SolarWinds software after those that knew have long left the scene.

And then, they have to rebuild on the assumption that everything has been compromised.

Note: Solarwinds has no CISO role.

SpaceLifeForm December 15, 2020 9:00 PM

Nicely summarizes SolarWinds reputation now.
(the CVEs listed are not complete)


Questonaut December 15, 2020 10:01 PM

There’s always much ambiguity when one has to admit to being snookered. We have great freedom of innovation with respect to a variety of ways of implementing applications, etc. I worry, so many ways of innovating. While I wouldn’t stifle a free market answer, we might need to weigh in on also believing a market heresy is innovative when we circumscribe a security understanding. Risking opposition to so many damned ways of doing anything. It slips through, doesn’t it?

SpaceLifeForm December 15, 2020 10:16 PM

@ Clive

Interesting observation from a SolarWinds user.

Multiple instances, not all backdoored per known hashes.
But, all kept up-to-date.

Hmmm. Can the backdoor recreate itself as a clean version after doing it’s recon?

Maybe, it becomes decided that the target is low hanging fruit, so just disappears.

But, if not low hanging fruit, perhaps establishing persistence in some other way, and then disappears?

If so, I would no longer even trust the hardware.


SpaceLifeForm December 15, 2020 11:26 PM

@ Clive, ALL

Oppsie. Silver Lake rings many bells.

They must have connected the dots when FireEye said they were hacked on December 7. Solarwinds knew they had the backdoor in place at FireEye.
SolarWinds knew that FireEye would find them.
SolarWinds knew the gig was up.


Silver Lake, a Silicon Valley investor with a history of high-profile tech deals including Airbnb, Dell and Twitter, sold $158 million in shares of SolarWinds on Dec. 7 — six days before news of the breach became public. Thoma Bravo, a San Francisco-based private equity firm, also sold $128 million of its shares in SolarWinds on Dec. 7.

Together, the two investment firms own 70 percent of SolarWinds and control six of the company’s board seats, giving the firms access to key information and making their stock trades subject to federal rules around financial disclosures.

Three executives from Thoma Bravo are directors on the SolarWinds board: Seth Boro, James Lines and Michael Hoffman. Three executives from Silver Lake — Kenneth Hao, Michael Bingle and Mike Widmann — are also on the board.

[don’t forget the bit-flipping router]

Clive Robinson December 15, 2020 11:52 PM

@ SpaceLifeForm,

That “new view” meme, reminds me of a cartoon in Private Eye many many years ago.

To get it though you need to know that “HMS Hermes” was a UK Navy “carrier”[1].

Also at the time there was what was considered a STD (now STI) Pandemic[2]. Back then people with the disease were refered to as “carriers”.

The cartoon was of a “Navy Day” dockside in front of an aircraft carrier and a slightly seedy/scruffy sailor was “propositioning” a young lady in a short tight dress.

She had a look of distaste on her face and on his cap band you could read “HMS Herpes”.



Clive Robinson December 16, 2020 12:20 AM

@ SpaceLifeForm,

Microsoft now has sinkholed the C2 domain.

Which one?

As I understand it from the Volexity report, “Dark Halo” had been buying numerous old domains up to help avoid detection. Also that “Dark Halo” also only used the domains just once for C2 activities.

Thus Microsoft may be “shutting the paddock gate” to the sound of distant hoof beats on that one, when the entire “string” has bolted…

With regards,

But, if not low hanging fruit, perhaps establishing persistence in some other way, and then disappears?

I’ve not seen enough information to say.

However the Volexity report does talk about “Dark Halo” not just cleaning up after establishing alternative access, but having a lot of stuff in RAM which disapears after a server reset / power cycle. So in this case it could be that “no evidence of hack” is in reality “Evidence of probable ownership” of your system by Dark Halo…

With regards the “insider trading” by selling shares, I’m not surprised, and as long as the SEC “sits on it’s thumbs” such sell offs prior to security breach publication are just going to get both more blatent as well as more numerous…

Clive Robinson December 16, 2020 12:20 AM

@ JimBob,

Whilst your statment is true…

The atribution to Russia you give befor may not be…

The supposed link to CosyBear apt28/29 etc, is not currently supported by either FireEye or Volexity, who call the attackers “UNC2452” and “Dark Halo” respectively. And say that they are unknown not just in attack style but who they work for or where they work from…

Hopefully time will reveal a little more publicly but it might not be for years (remember Stuxnet, and after years the admission that the target had realy been North Korea?).

SpaceLifeForm December 16, 2020 10:22 AM

@ Clive

Sinkholed: avsvmcloud [dot] com

Note 2018 date


The primary domain is interesting, as it first appears long before the Spring 2020 timeframe when SUNBURST is believed to have started (Spring 2020), in July 2018. A closer view of Whois history via DomainTools Iris shows a noticeable change in the domain in late 2019:


The domain shifted to GoDaddy parking infrastructure on 20 December 2019, roughly the same time as the change in registration.

The domain remains on GoDaddy parking infrastructure until 27 February 2020, when it shifts to Microsoft cloud hosting (52.171.135[.]15, then 13.65.251[.]83)

At the same time hosting shifts to Microsoft, the domain changes authoritative name server to self-hosting at Avsvmcloud[.]com.

SpaceLifeForm December 16, 2020 11:03 AM

@ Clive, ALL

Article on the sinkhole


SpaceLifeForm December 16, 2020 2:19 PM

@ Clive, ALL

As I suspected, the backdoor could disappear.


As part of our analysis of SUNBURST, we identified a killswitch that would prevent SUNBURST from continuing to operate.

This killswitch will not remove the actor from victim networks where they have established other backdoors.

ResearcherZero December 16, 2020 8:40 PM

Only 99 domains to go… :p

After the DHS Inspector Generals report from 2018, DHS management said “corrective actions” were underway.

Then the 2019 DHS Inspector Generals report came out.

“After a decade of negligence, our federal agencies have failed at implementing basic cybersecurity practices, leaving classified, personal and sensitive information unsafe and vulnerable to theft,”

You can read about the improvements here. At least we know that some agencies were applying updates this year.

SolarWinds have removed their partial customer list from their website, which of course is on the internet.

ResearcherZero December 16, 2020 9:02 PM

@Clive Robinson

We know exactly who they work for, we just don’t give out that kind of information.

ResearcherZero December 16, 2020 9:42 PM

Which is a pity because the public could really do with some information from time to time about who made who.

Adrift in a world of my own (ooh ooh)
I play the game but to my real shame
You’ve left me to dream all alone
Too real is this feeling of make believe

these three ‘outsiders’

Just laughing and gay like a clown (ooh ooh)
I seem to be what I’m not (you see)
I’m wearing my heart like a crown
Pretending that you’re still around

Those new F35s look beautiful flying overhead, and boy are they fast.
Thank you very much.

SpaceLifeForm December 16, 2020 10:27 PM

Appears that main development and build system of SolarWinds software is in Brno, Czech Republic.

Someone was very, very familiar with the source code and had control of their SCM.

Either SolarWinds had been owned for sometime, or it was an inside job.


Here is stuff that maybe points to inside job.
The backdoor would check if it was on internal networks and not run. The domain names were known, but hashed in the backdoor code.


Clive Robinson December 17, 2020 1:08 AM

@ SpaceLifeForm,

Either SolarWinds had been owned for sometime, or it was an inside job.

Reading through the Reversing Labs blog article you find,

“Such string obfuscation is repeated throughout the code. And that’s the balance between standing out in a software developer review and fooling the security systems, a gamble that has paid off for the attackers.”

The attackers have taken quite a lot of involved steps to remain covert in other places, so “Why take a gamble?”…

Especially when they appear to also have near unlimited access to the sourcecode, to watch authorised changes by SolarWinds staff in near real time…

Which brings me back to the point that it’s an insider, to the company at a sufficiently high level that “gambles” are not necessary, they can be “talked around”.

I guess the next stage for a SolarWinds internal investigation is to see who put the changes in the source control system and when, though how reliable that might be is another question.

It brings up shades of the way Ed Snowden collected his trove by using other peoples accounts/credentials.

When you look down the megabets thread you come across Jim Sykora’s tweet,

“Wouldn’t the best ways to gain this detailed internal info be a) prolonged undetected access to SolarWinds internal networks to learn this or b) paying/subverting a SW employee?”

Did not expand the implications of his point and walk up the managment ladder a bit…

Look at it this way, if the company is doing things even half way sensibly, their ICTsec team would be looking for unusual activities on the internal networks. This would involve instrumenting the internal networks. So an external attacker going into the network blind is likely to hit an Intrusion Detection System they can not see as they enumerate the network. However a low level insider not in the ICTsec department would have only a limited knowledge of the network thus would probably bump an IDS as well. However someone who managed the development teams would have a much wider scope and would probably be aware of both the development and test networks as well as any IDS systems as the teams underneath them would trip them from time to time…

Either the attackers got very very lucky or they had eyes inside at a sufficiently high level. As the old saying goes,

“I believe in luck, and I create as much of it as I can”.

This sort of luck is not “blind luck” but has better than 20-20 vision, and they are not “counting cards” but “dealing them off the bottom”. The only question is how they know what’s on the bottom of the deck out os sight and supposadly unguessable…

JonKnowsNothing December 17, 2020 10:40 AM

@Clive @SpaceLifeForm @All

This operation is so deep it reminds me of the story about The Trust and how that ensnared a lot of high level folks 1921-1926.

And it wasn’t even computer based.

ht tps://
(url fractured to prevent autorun)

Clive Robinson December 17, 2020 12:56 PM

@ JonKnowsNothing, SpaceLifeForm, ALL,

This operation is so deep it reminds me of the story about The Trust

The problem with “The Trust” was that it was more smoke and mirrors than reality.

I don’t know if you saw any of the original three Harry Palmer films, but one “The Billion Dollar Brain” was a fascinating example of art imitating life, but years before life happened. In a way it was Science Fiction rather than a tale of espionage.

But in a way The Trust was an early example, of plots within plots, intrigue layered on intrigue, with no actor knowing what part they were playing nor on who’s side they were on officially or not, nor importantly what the outcome or their destiny would be.

From this respect SolarWinds is definately shaping up to be atleast a couple of bowls of popcorn entertainment. Now all we need is a crackling fire and a good bottle of whisky to help it all along on these cold windy nights 😉

SpaceLifeForm December 17, 2020 1:43 PM

OpSec, other backdoors.

All of the hardware is un-trustable, even if it never had any SolarWinds software on it. If on the LAN, even I would not trust a desktop that was on that LAN if there was any possible lateral or hopscotch movement from a SolarWinds server machine.



CISA is aware of compromises, which began at least as early as March 2020, at U.S. government agencies, critical infrastructure entities, and private sector organizations by an APT actor. This threat actor has demonstrated sophistication and complex tradecraft in these intrusions. CISA expects that removing the threat actor from compromised environments will be highly complex and challenging. This adversary has demonstrated an ability to exploit software supply chains and shown significant knowledge of Windows networks. It is likely that the adversary has additional initial access vectors and tactics, techniques, and procedures (TTPs) that have not yet been discovered.

[Skiping SAML and impossible logins]

Operational Security

Due to the nature of this pattern of adversary activity—and the targeting of key personnel, incident response staff, and IT email accounts—discussion of findings and mitigations should be considered very sensitive, and should be protected by operational security measures. An operational security plan needs to be developed and socialized, via out-of-band communications, to ensure all staff are aware of the applicable handling caveats.

SpaceLifeForm December 17, 2020 2:23 PM

@ Clive, JonKnowsNothing, ALL

Parse carefully what I bolded above.

What CISA is telling you is that you can not trust any existing Internet Protocol for commumnications.

You do know who this APT/TLA is, right?


You better figure it out if you do not know.

SpaceLifeForm December 17, 2020 2:55 PM

@ Clive, JonKnowsNothing, ALL

One more point if needed.

If your org finds IOC, yes, contact FBI and/or CISA.


Arrange to discuss OFFLINE.

Clive Robinson December 17, 2020 3:25 PM

@ SpaceLifeForm

“It is likely that the adversary has additional initial access vectors and tactics, techniques, and procedures (TTPs) that have not yet been discovered.”

Including those that are “connected” but not of necessity by the Internet at the first and second party nodes (In the UK for instance all phone calls cross on an IP switched backbone even inside local of premises exchanges).


Means : Never Say Anything is on line or by other “connected” method, as not just terminal but all equipment in between can be considered not just vulnerable but exploited and some of those exploits have leaked yet again and advantage has been taken by unexpected and apparently unknown players.

As I’ve mentioned in the past “air gapping is not enough”… Energy carries the message in many forms both conducted and radiated channels.

Which makes life fun for some, after all it’s not as though we have not been warning people here for more than a decade…

I wonder if this might turn out to be another “Ed Snowden” moment…

SpaceLifeForm December 17, 2020 3:45 PM

@ Clive, JonKnowsNothing, ALL

But in a way The Trust was an early example, of plots within plots, intrigue layered on intrigue, with no actor knowing what part they were playing nor on who’s side they were on officially or not, nor importantly what the outcome or their destiny would be.

Sounds like Compartmentalization.

Today, you must ask: who controls Ultra?

Parse above.

SpaceLifeForm December 17, 2020 3:56 PM

@ Clive


Means : Never Say Anything is on line or by other “connected” method, …

Should read:

Means : Never Say Anything if on line or by other “connected” method, …

SpaceLifeForm December 17, 2020 9:06 PM

@ Clive, JonKnowsNothing, ALL

Not sure I would have written the URL this way, but anyway, this is the headline:

Microsoft says it found malicious software in its systems


“Like other SolarWinds customers, we have been actively looking for indicators of this actor and can confirm that we detected malicious Solar Winds binaries in our environment, which we isolated and removed,” a Microsoft spokesperson said, adding that the company had found “no indications that our systems were used to attack others.”

SpaceLifeForm December 17, 2020 10:24 PM

@ Clive, JonKnowsNothing, ALL

So, he wants to believe that if there are no identified problems, all is fine.

Maybe everything is still functioning as expected. At this time.

But, it is dumb on his part to believe that there will not be future problems considering the amount of time for exfiltration.

Maybe he is only worried about the stock price.


Microsoft president Brad Smith says Reuters report is false. “We have no indication of this.” Microsoft stands by Sunday statement: “We also want to reassure our customers that we have not identified any Microsoft product or cloud service vulnerabilities in these investigations.”

Clive Robinson December 18, 2020 1:05 AM

@ SpaceLifeForm, ALL,

‘… a Microsoft spokesperson said, adding that the company had found “no indications that our systems were used to attack others.”…’

That is a curious statment at best, saying more by what it does not say than it does in words due to it’s ambiguity.

Let us assume in Microsoft’s case it was an “outsider attack”. To do the same sort of attack alledged against SolarWinds an initial four step approach is needed,

1, Gain entry undiscovered.
2, Enumerate systems without tripping IDSs and the like.
3, Gain access to development systems.
4, Gain access to source code.

Once they have the source code they can start to work on it in various ways and gain intel that is not available from compiled code.

For instance variable names give insight as to how the programmer views their use, as do method names etc. But it also helps build a view of the direction the code is taking and which parts for what ever reason have received little attention. Also the career trajectory of programers and what their expertise may be.

And a lot more besides, look on it as “traffic analysis” of “code development” in the right hands such knowledge would be highly profitable in many ways most of us would not realise unless we spent some time “thinking hinky” as @Bruce might put it.

This is the sort of analysis I would expect from the upper end of Level III attackers such as the UK GCHQ and US NSA, and many Western European nations SigInt agencies. Such analysis is like the interest on an investment, it starts small and grows large depending on the quality of the investment.

So just having had access to Microsoft’s source code will have caused some harm to Microsoft’s customers down the line.

I could go on down the corruption of the developer chain showing many other potential “harms in waiting” that the Microsoft comment does not cover, and none of them are good.

However if I do, each step gets closer to becoming a DIY Guide[1] for attackere, and I’ve been told off for that in the past. So I’ll let others have their own thoughts on that.

[1] For those that want to know my early thinking on why code signing is such a bad idea, have a look on this blog pre-stuxnet by a few years. @Nick P, myself and one or two others had several conversations on the matter and pointed out a number of attack vectors some of which later appeared in Stuxnet a decade ago.

Clive Robinson December 18, 2020 2:15 AM

@ SpaceLifeForm, JonKnowsNothing, ALL,

Not sure I would have written the URL this way, but anyway, this is the headline

It’s not just the headline in that Reuters artical…

Did you read down towards the bottom and what was worrying Senetors?

Lets just say,

“Once a crook always a crook”

Only they don’t want people knowing just how much of a “corkscrew” they realy are, even though the IRS does…

Leave a comment


Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via

Sidebar photo of Bruce Schneier by Joe MacInnis.