More on the SolarWinds Breach

The New York Times has more details.

About 18,000 private and government users downloaded a Russian tainted software update—­ a Trojan horse of sorts ­—that gave its hackers a foothold into victims’ systems, according to SolarWinds, the company whose software was compromised.

Among those who use SolarWinds software are the Centers for Disease Control and Prevention, the State Department, the Justice Department, parts of the Pentagon and a number of utility companies. While the presence of the software is not by itself evidence that each network was compromised and information was stolen, investigators spent Monday trying to understand the extent of the damage in what could be a significant loss of American data to a foreign attacker.

It’s unlikely that the SVR (a successor to the KGB) penetrated all of those networks. But it is likely that they penetrated many of the important ones. And that they have buried themselves into those networks, giving them persistent access even if this vulnerability is patched. This is a massive intelligence coup for the Russians and failure for the Americans, even if no classified networks were touched.

Meanwhile, CISA has directed everyone to remove SolarWinds from their networks. This is (1) too late to matter, and (2) likely to take many months to complete. Probably the right answer, though.

This is almost too stupid to believe:

In one previously unreported issue, multiple criminals have offered to sell access to SolarWinds’ computers through underground forums, according to two researchers who separately had access to those forums.

One of those offering claimed access over the Exploit forum in 2017 was known as “fxmsp” and is wanted by the FBI “for involvement in several high-profile incidents,” said Mark Arena, chief executive of cybercrime intelligence firm Intel471. Arena informed his company’s clients, which include U.S. law enforcement agencies.

Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123”

“This could have been done by any attacker, easily,” Kumar said.

Neither the password nor the stolen access is considered the most likely source of the current intrusion, researchers said.

That last sentence is important, yes. But the sloppy security practice is likely not an isolated incident, and speaks to the overall lack of security culture at the company.

And I noticed that SolarWinds has removed its customer page, presumably as part of its damage control efforts. I quoted from it. Did anyone save a copy?

EDITED TO ADD: Both the Wayback Machine and Brian Krebs have saved the SolarWinds customer page.

Posted on December 17, 2020 at 2:18 PM26 Comments

Comments

JonKnowsNothing December 17, 2020 3:41 PM

@All

re:Wayback Machine as archive

fwiw: Do not rely on the WBM as a persistent safe source of archive material. The WBM can and does delete information regularly and there are lots of conditions where the archive is required to do so.

If you find something of interest – make a copy if that’s safe for you to do so. Sometimes it is not safe depending on the context of the page.

What is definitely not safe, is to believe the WBM holds all archives of the entire internet for ever and ever. (1) It doesn’t and cannot.

1, A multiple lifetime storage system is in Bluffdale UT owned by the NSA. One might expect that there are similar archives owned by other like minded groups.

ht tps://www.theguardian.com/politics/2013/nov/13/conservative-party-archive-speeches-internet

The Conservatives have removed a decade of speeches from their website and from the main internet library…In a remarkable step the party has also blocked access to the Internet Archive’s Wayback Machine…

(url fractured to prevent autorun)

Clive Robinson December 17, 2020 4:02 PM

@ ALL,

Please note that neither FireEye who first broke the news nor Volexity say “Russian” they are both at pains to distance themselves currently.

Thus I’d keep a careful eye on this it may well turn out to be a lot more interesting.

If you look you will see that the claims do not indicate it is APT28 or APT29 as there is no similarity or tracability in the tools and techniques used by the attackers. So far they appear to be novel to “Dark Halo”.

The problem is that it appears to be way to many novel tools and techniques at one time, and levels of sophistication above that of quite a few Nation State (Level III) actors.

Which might mean that either they have been around for some time unnoticed or they aquired tools and techniques from others that have been unnoticed for some time.

It has in the past been suggested that US AV companies remove detection for attacks they believe to be from the likes of the Five Eyes SigInt agencies… Thus the tools and techniques might have originated from these organisations.

Some have even suggested that what has been exploited was another “backdoor” not to disimilar in idea to those found in RSA and Jupiter products. However such SigInt tools and techniques were very much more subtle than what has happened to SolarWinds. In fact the “backdoor” so far described is more like the sort of thing the FBI has been getting Universities to build for them to attack TOR users or with the assistance of the DoJ try and force equipment suppliers like Apple to install using archaic laws.

Much of the sophistication claimed for “Dark Halo” is predicated on the fact that these attackers are “external actors” from begining to end.

However if instead you assume they only found and used a preexisting backdoor then their attack becomes much more believable. Hence their reason togo after the FireEye Red Team Tools. Which they would surely not need if they were as sophisticated as some have indicated.

Either way we have a puzzle here and many of the pieces are missing and it’s going to become more intriguing with time… Speaking of which,

Then there is the behaviour of major investors “bailing” by what is probably “insider trading” why would they do that and in effect take a loss? I guess they are not going to say and will use any leverage they have at hand to kill any prosecution before it starts. So it’s worth keeping an eye on them as well.

So keep your ears abd eyes open and the popcorn on the go, this story has the feeling it’s going to get some very interesting legs real soon.

SpaceLifeForm December 17, 2020 4:22 PM

@ Clive, JonKnowsNothing, ALL

It is important to observe that the WBM page archive is just a very small subset of the SolarWinds clients.

Very small.

SpaceLifeForm December 17, 2020 4:59 PM

@ Clive, JonKnowsNothing, ALL

Note: FTP Upload access does not mean that the source code magically made it into the build system.

That would require an inside job.

Unless, owned for a long time.

And software was in place to detect new upload, insert into SCM, and remove from FTP folder.

This is not trivial.

And it also would require a worthless POS SCM/Build System that has no tracking, no signoffs.

Everything points to inside job.

JonKnowsNothing December 17, 2020 5:37 PM

@SpaceLifeForm

re:magic in the build

I can think of several ways source code can be added to a build system and no one would notice.

One way is when there are several devs working on the same project. Source can be added and everyone else presumes it’s the other dudettes code. As long as the merge is seamless and doesn’t crack up another hunk the code would be added to the build.

This presumes that the attacker can access the source base and run compiles to be sure nothing cracks but it would leave the code behind to read.

The other option is to get an obj file into the linker pool. A good number of builds only rebuild new code to reduce build time. If the obj were a separate file then it would be reviewable if noticed. However, if the bad-obj was merged or linked to a good-non-changing obj it could be unnoticed for a long time.

As to Source Control Systems – Back when most were company hosted, there were some not that good and charged mega $$$. Some years ago I had an encounter with such a beast. Surprising given the nature of the development not even passwords or branch restrictions – just tarball up the entire source code for a mega-mega-mega corporation. One might hope they changed it but I wasn’t holding my breath at the time. No one on the project saw the least teeny bit of a problem and the devs didn’t want any more restrictions or validations.

If the insider-access started in that time frame, it would have been ezpz to do.

The interesting aspect is: there are no obvious traces…

John Carter December 17, 2020 6:45 PM

Hah!

Too much truth in advertizing!

From the front page of their web site….

“IT Service Management without the friction”

ResearcherZero December 17, 2020 9:31 PM

@John Carter

No friction at all. You have to login in now to see the support page, but the following quote still brings up the page in search results.

“To run SolarWinds products more efficiently, you may need to exclude certain files, directories and ports from anti-virus protection and GPO restrictions.”

hxxps://support.solarwinds.com/SuccessCenter/s/article/Files-and-directories-to-exclude-from-antivirus-scanning-for-Orion-Platform-products?language=en_US

“We also list the service accounts that should be added for optimal performance and to allow all Orion products to access to required files with required permissions.”

It’s a stroke of genius, worth every cent. This is what makes the whole ‘backdoor’ and ‘ghost user’ encryption advice from government lawyers, with absolutely no security training or interest, so brilliant. If you can’t see it, and you don’t know about it, it didn’t happen.

“Buy our radium , it will cure anything.”

The more than 800 charges laid and children rescued in the following investigation was due to a lot of painstaking work. Investigations take years of work, and a willingness to make them happen. It is the willingness that is the greatest problem, and there are no magic fixes that will let anyone escape that.

hxxps://www.dw.com/en/australia-police-expose-global-child-sex-abuse-ring/a-55560115

ResearcherZero December 17, 2020 10:33 PM

The most frustrating thing is the lack of accountability. The political spin that attacks are sophisticated, is nothing more than an attempt to disguise the fact that legislation to secure government systems has been stalled in the house for close to a decade. Instead of anything useful, it has instead turned into a partisan farce, that must have the adversaries of the United States rolling on the floor laughing.

Even when an act manages to pass, you still have to get departments in each and every state to implement basic security requirements.
If you read audits from state agencies from nations around the world who have managed to enact legislation, the state of cybersecurity in most state departments is appalling. The audits are often the same year after year.
No auditing, worst ever passwords, missing logs…

A hacker does not need an IQ of 197 and 15% of your password, a low to average IQ is quite enough if they can guess your password, and there is no auditing.

I have read department cybersecurity audits for years and thought to myself “my @#$%ing God, my @#$%ing God, why are things not way way worse already”, and “holy @#$%ing 5-it, holy @#$%ing 5-it”. (because they are and nobody knows because of the lack of proper auditing)

You wonder to yourself why you even bothered preparing and submitting assessments, and reporting security vulnerabilities when they are just left unfixed and exploitable a year later. Then you remember that one rather large corporation changed the FTP credentials that one time many years ago, perhaps it’s possible again. Perhaps you don’t have to knock over a server and bring down an entire network to get someone to fix something.

…Perhaps the support staff are not technically illiterate, completely overconfident, and won’t open an attachment that says “password protected, DO NOT OPEN on the network”.

Maybe someone will listen to the administrator. Maybe the administrator won’t shoot themselves after a prolonged and severe breakdown.

ResearcherZero December 18, 2020 1:04 AM

A “do over” is mandatory and entire new networks need to be built — and isolated from compromised networks…
hxxps://www.nytimes.com/2020/12/16/opinion/fireeye-solarwinds-russia-hack.html
Cyber threat hunters that are stealthier than the Russians must be unleashed on these networks to look for the hidden, persistent access controls…
This will be difficult work as the Russians will be watching every move on the inside…

They found suspicious activity in networks belonging to the Federal Energy Regulatory Commission (FERC), Sandia and Los Alamos national laboratories in New Mexico and Washington, the Office of Secure Transportation and the Richland Field Office of the DOE.
hxxps://www.politico.com/news/2020/12/17/nuclear-agency-hacked-officials-inform-congress-447855

Clive Robinson December 18, 2020 1:27 AM

@ SpaceLifeForm, JonKnowsNothing, ALL,

It is important to observe that the WBM page archive is just a very small subset of the SolarWinds clients.

There are various figures floating around. As far as I can tell the important ones are,

300,000 Customers.
18,000 Downloaded the backdoor.
~100 Select customers attacked.

With atleast one report indicating 180k rather than 18k having downloaded the backdoor.

If the 300,000 customers is true, then it is likely that is “world wide customers”. If the 18,000 downloaded is true that is actually very surprisingly small. That is at only 6% of customers you would have to ask “Why so few?”, 60% would be more likely unless there was a mechanism somewhere that “selected” who did and did not get the “backdoor”.

Just explaining this discrepancy alone might reveal some major implications, but also tells you just how little information we have publically[1].

I get the feeling many want the public to just swallow the “It’s wot Russkies dun” line so they can hide one heck of a pile of nastyness out of sight ubder the corner of the rug.

I suspect it’s going to take some time to get to the truth of the matter, and I don’t think people are going to like the truth, so it’s showing all the signs of something that is going to be kicked very hard into the long grass with a good downhill slope to keep it rolling away far far out of sight…

[1] publicly / publically, both are correct but which to use?

Clive Robinson December 18, 2020 1:46 AM

@ SpaceLifeForm, JonKnowsNothing, ALL,

Note: FTP Upload access does not mean that the source code magically made it into the build system.

No, but more importantly it’s also true of the other end of the chain. That is uploading an executable/interpreted code file does not mean it will be downloaded and run by a client machine.

It would need access to a lot more than just the code signing key…

For either of these things a lot would have to be wrong with SolarWinds development, test, maintainence etc processes, seriously wrong… Or a sufficiently high level person in SolarWinds Development managment being in the know and shuffling things through and keeping a lid on any internal questions arising.

The questions that investigators should be asking are effectively “Which?” and “Why?”. And you can bet your bottom dollar it’s not just Venture Capitalists that “Want to be the heck out of Dodge” before anyone asks those questions.

Mikko Kiviranta December 18, 2020 3:16 AM

No manufacturers of voting machines in the list by Brian Krebs, fortunately.

I was just wondering whether the surprisingly wide support to the trumpist ideas in the elections was not a surprise after all. In Europe there is widespread suspicion whether US can be relied on as a long-term ally, if there is a large fraction of voting populace wishing to re-play the previous four years. It would be beneficial to many adversaries of the US to widen the gap.

It seems voting machines were not affected, after all. Many of those don’t leave a paper trail, unlike postal voting which does.

keiner December 18, 2020 7:52 AM

Let’s assume for a second, one of my banks is on the list and this bank has “TAN via SMS” as the (nearly) only security for transfering money from my account.

If somebody stole the phone number for this SMS and got a new SIM card for this number… Just saying…

jones December 18, 2020 9:19 AM

@JonKnowsNothing

Indeed, the “dodgy dossier” in .doc format was removed from the archive.org page on the 10 Downing Street website where it had been posted. The .doc file could be inspected for metadata, like who actually put the thing together…

JonKnowsNothing December 18, 2020 11:24 AM

@Clive @All

re: High Level Insider

@Clive:

a lot would have to be wrong with SolarWinds development, test, maintainence etc processes, seriously wrong… Or a sufficiently high level person in SolarWinds Development managment being in the know and shuffling things through and keeping a lid on any internal questions arising.

It’s happened before: Yahoo!, M. Mayer then CEO Yahoo!, authorized it herself privately.

ht tps://en.wikipedia.org/wiki/Alex_Stamos#Yahoo!

In 2014, Stamos joined Yahoo! as CSO.While at Yahoo!, he testified to Congress on online advertising and its impact on computer security and data privacy. He publicly challenged NSA Director Michael S. Rogers on the subject of encryption backdoors in February 2015 at a cybersecurity conference hosted by New America. He resigned in June 2015 in response to a then-undisclosed program to scan incoming email on behalf of United States government intelligence agencies .

ht tps://en.wikipedia.org/wiki/Marissa_Mayer

(url fractured to prevent autorun)

DaveR December 18, 2020 3:23 PM

@ResearcherZero – in the reply to John Carter

Regarding the link to SolarWinds recommended exclusions – I see the problem with the company in the path. They apparently refer to their support group as the “Success Center”.

I’ve worked too many years now as a System Administrator and I really shudder when I see that a company I deal with has changed the name for the support group to some sort of “success center”. Generally the change seems to imply that more emphasis will be placed on “feel good” activity over providing real support. Customer Success Centers should be viewed as a reason to pass over a company for consideration, in my opinion. (Don’t even get me started on training videos taking the place of real documentation! 🙂 )

I tried to visit the link, but like most “modern” wesites, nothing at all was displayed without java script. Considering the recent activity I decided it was best to not allow any of it.

xcv December 18, 2020 7:12 PM

Key concepts straight out of NSA’s press release

… This advisory provides guidance to National Security System (NSS), Department of Defense (DoD), and Defense Industrial Base (DIB) network administrators to detect and mitigate against malicious cyber actors who are manipulating trust in federated authentication environments to access protected data in the cloud. It builds on the guidance shared in the cybersecurity advisory regarding VMware with state-sponsored actors exploiting CVE 2020-4006 and forging credentials to access protected files, though other nation states and cyber criminals may use this tactic, technique, and procedure (TTP) as well. …

Government employees are lazy and entrenched behind their desks at work at the NSA, and while they speak in a low whisper, the truth is that they relly aren’t much better at keeping secrets than any other government agency with a communal office environment given to “the usual” infighting, backstabbing, gossiping, chit-chat, and office politics, not to mention the Title IX sexual harassment lawsuit jackpots with the requisite “hostile work environment” and the classified information charges against Reality Winner.

Kerberos: The Network Authentication Protocol of course is one such “federated authentication environment”, probably the one we are most familiar with in the FOSS world.

The phrase “Tactic, Technique, and Procedure” is probably a press-room misreading of the common acronym “TTP” for Trusted Third Party which in all likelihood is really what they are talking about in the context of a “federated authentication environment.”

xcv December 19, 2020 4:04 PM

I wonder if the NICS EZ-Check FBI universal instant employment background check system was hacked or compromised as part of the breach.

What do we do when made-up arrests and court convictions we never heard about suddenly start showing up in the system when we’re looking for employment or trying to land a contract?

ResearcherZero December 20, 2020 10:45 PM

@xcv
State registries for the justice system are notorious for lack of auditing and oversight, and police departments are a favorite target for organizations like the GRU. No one actually checks that ID of people who make statements, no one checks that people who appear in court are who they claim to be, and no one checks that the documents on court and police systems are actually correct.

Once prosecutors are compromised, it is really hard to get convictions against foreign actors working as police officers or detectives, as they are legally allowed to lie as part of their job. Most importantly no one is willing to take responsibility for just how bad they screwed up and ignored repeated warnings, then allowed it continue for decades, and then it blows up in everyone’s face…

Though, this is just not a problem in the United States, it’s a very widespread and active problem. It’s sort of like everyone is sitting on a create of dynamite, that’s sweating nitro, and they are all smoking crack.
But don’t worry, we fixed the problem of being able to shoot down ICBMs, made ’em hyper-sonic.

The report, commissioned by Senator Ben Cardin, D-MD and numbering more than 200 pages, directly criticises President Trump for failing to respond to the threat
hxxps://www.foreign.senate.gov/press/ranking/release/cardin-releases-report-detailing-two-decades-of-putins-attacks-on-democracy

on August 24, 2016, the Russian hackers sent spoofed emails purporting to be from Google to employees of an unnamed U.S. election software company
(classified NSA report – on the public record)
hxxps://assets.documentcloud.org/documents/3766950/NSA-Report-on-Russia-Spearphishing.pdf

the US Intelligence Community concluded Russia was responsible
hxxps://www.dni.gov/files/documents/ICA_2017_01.pdf

xcv December 20, 2020 11:59 PM

@ResearcherZero

State registries for the justice system are notorious for lack of auditing and oversight, and police departments are a favorite target for organizations like the GRU. No one actually checks that ID of people who make statements, no one checks that people who appear in court are who they claim to be, and no one checks that the documents on court and police systems are actually correct.

Once prosecutors are compromised, it is really hard to get convictions against foreign actors working as police officers or detectives, as they are legally allowed to lie as part of their job. Most importantly no one is willing to take responsibility for just how bad they screwed up and ignored repeated warnings, then allowed it continue for decades, and then it blows up in everyone’s face…

Though, this is just not a problem in the United States, it’s a very widespread and active problem. It’s sort of like everyone is sitting on a create of dynamite, that’s sweating nitro, and they are all smoking crack.

Just not or not just? Yes it is a very real problem. Domestic as well as forign enemies have used the U.S. court system as an arbitrary tool to pick on me, revoke my rights, harass me, slander my name, set me up for false arrests and false criminal charges, and so on and so forth, over and over again, world without end.

But don’t worry, we fixed the problem of being able to shoot down ICBMs, made ’em hyper-sonic.

The report, commissioned by Senator Ben Cardin, D-MD and numbering more than 200 pages, directly criticises President Trump for failing to respond to the threat
hxxps://www.foreign.senate.gov/press/ranking/release/cardin-releases-report-detailing-two-decades-of-putins-attacks-on-democracy

on August 24, 2016, the Russian hackers sent spoofed emails purporting to be from Google to employees of an unnamed U.S. election software company
(classified NSA report – on the public record)
hxxps://assets.documentcloud.org/documents/3766950/NSA-Report-on-Russia-Spearphishing.pdf

the US Intelligence Community concluded Russia was responsible
hxxps://www.dni.gov/files/documents/ICA_2017_01.pdf

Weapons system technology is all well and good, but it doesn’t do anything to fix our terminally broken unconstitutional system of punishment and debt collection without due process, such as it exists in the United States, from SCOTUS on down, without defense, without recourse, without redress, and without possibility of appeal.

Meanwhile the Democrats are intent on throwing out the baby with the bathwater in Russia, even as they got away with foul play in the U.S. court system. That is unnecessary. Various factions in Russia do present a threat, but it’s the old-guard hard-line Communist Party threat from the days of the KGB before it became known as the FSB — Russia has a lot of internal conflict, and America will have to continue cooperating with the friendlier parties, groups, organizations, and individuals there.

Security Sam December 21, 2020 11:40 AM

SolarWinds with nefarious currents
Produce ripples in laminar flows
Accessing data streaming in torrents
Provoking ire and coming to blows.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.