Latest on the SVR’s SolarWinds Hack

The New York Times has an in-depth article on the latest information about the SolarWinds hack (not a great name, since it’s much more far-reaching than that).

Interviews with key players investigating what intelligence agencies believe to be an operation by Russia’s S.V.R. intelligence service revealed these points:

  • The breach is far broader than first believed. Initial estimates were that Russia sent its probes only into a few dozen of the 18,000 government and private networks they gained access to when they inserted code into network management software made by a Texas company named SolarWinds. But as businesses like Amazon and Microsoft that provide cloud services dig deeper for evidence, it now appears Russia exploited multiple layers of the supply chain to gain access to as many as 250 networks.
  • The hackers managed their intrusion from servers inside the United States, exploiting legal prohibitions on the National Security Agency from engaging in domestic surveillance and eluding cyberdefenses deployed by the Department of Homeland Security.
  • “Early warning” sensors placed by Cyber Command and the National Security Agency deep inside foreign networks to detect brewing attacks clearly failed. There is also no indication yet that any human intelligence alerted the United States to the hacking.
  • The government’s emphasis on election defense, while critical in 2020, may have diverted resources and attention from long-brewing problems like protecting the “supply chain” of software. In the private sector, too, companies that were focused on election security, like FireEye and Microsoft, are now revealing that they were breached as part of the larger supply chain attack.
  • SolarWinds, the company that the hackers used as a conduit for their attacks, had a history of lackluster security for its products, making it an easy target, according to current and former employees and government investigators. Its chief executive, Kevin B. Thompson, who is leaving his job after 11 years, has sidestepped the question of whether his company should have detected the intrusion.
  • Some of the compromised SolarWinds software was engineered in Eastern Europe, and American investigators are now examining whether the incursion originated there, where Russian intelligence operatives are deeply rooted.

Separately, it seems that the SVR conducted a dry run of the attack five months before the actual attack:

The hackers distributed malicious files from the SolarWinds network in October 2019, five months before previously reported files were sent to victims through the company’s software update servers. The October files, distributed to customers on Oct. 10, did not have a backdoor embedded in them, however, in the way that subsequent malicious files that victims downloaded in the spring of 2020 did, and these files went undetected until this month.

[…]

“This tells us the actor had access to SolarWinds’ environment much earlier than this year. We know at minimum they had access Oct. 10, 2019. But they would certainly have had to have access longer than that,” says the source. “So that intrusion [into SolarWinds] has to originate probably at least a couple of months before that ­- probably at least mid-2019 [if not earlier].”

The files distributed to victims in October 2019 were signed with a legitimate SolarWinds certificate to make them appear to be authentic code for the company’s Orion Platform software, a tool used by system administrators to monitor and configure servers and other computer hardware on their network.

Posted on January 5, 2021 at 6:42 AM48 Comments

Comments

Eitan Caspi January 5, 2021 8:51 AM

If this was not clear until today, then this case with Solarwinds makes it very clear – a vendor who digitally signs his files guarantees not only that they have not been modified since signing and that they originated from it, but mainly that it is responsible for their content.

If an attacker injects malicious code into the vendor’s code and therefore “rides” on the vendor’s code and the vendor “blindly” signs the code without checking the content – this is a full and clear responsibility for the failure of the vendor that hereby became an attack channel.

Clive Robinson January 5, 2021 9:06 AM

@ Eitan Caspi,

… a full and clear responsibility for the failure of the vendor that hereby became an attack channel.

True, but it appears that nearly all the biggest corps have been supply chaon attacked.

Which means one of two possibilities come to mind,

1, Code signing is of no use.
2, Auditing is not employed effectively.

Personally I know it’s both to varying degrees in most occasions I’ve looked at it.

You will I’ve been ssying code signing is fairly usless for years on this blog.

Goat January 5, 2021 9:24 AM

@Clive Robinson

Code signing reminds me of one thing that I would like to point is the cost of signing binaries in windows such that they may be verified. It is so expensive..

Clive Robinson January 5, 2021 9:40 AM

@ Goat,

… the cost of signing binaries in windows such that they may be verified. It is so expensive

It is and it actually does not achieve very much…

Basically all it does is to tell the loader that it can load the binary into RAM. But that’s it, it says nothing about the quality of the code, the security of the code, only that some person unknown has hashed it up and then signed it, not even who.

It also says nothing about what happens to it whilst in RAM, so gives no protection/security in the execution environment…

TimH January 5, 2021 9:45 AM

I wonder if the SolarWinds attack vector was known by or inserted by the US. So the shouting Russia! Russia! is partly misdirection away from detailed analysis of what exactly happened.

The statement “Some of the compromised SolarWinds software was engineered in Eastern Europe, and American investigators are now examining whether the incursion originated there, where Russian intelligence operatives are deeply rooted.” is of such ignorance on several levels that the statement must be targetting a general public audience.

Clive Robinson January 5, 2021 9:47 AM

@ Eitan Caspi,

I don’t think code signing is useless, just that is needs to be done with great care, attention and responsibility.

All of which has very real costs attached, that also slow down release dates and a whole other pile of disincentives that managment and marketing do not want.

All of which tends to suggest it won’t get done or will at best get “lip service” only.

The only upside for managment is it keeps the idea of a walled garden for profitability alive by keeping out competition…

Anon E. Moose January 5, 2021 10:26 AM

NYT Propagandize much?

“The hackers managed their intrusion from servers inside the United States, exploiting legal prohibitions on the National Security Agency from engaging in domestic surveillance and eluding cyberdefenses deployed by the Department of Homeland Security.”

This is blatant promotion of citizen surveillance. It is how we get blacklists, no fly lists and people attached to those lists without just cause.

Solarwinds may have been really bad with security, but that does not justify the NSA watching our movements.

David Glynn January 5, 2021 10:36 AM

Do we know whether this compromise of systems would have still been possible if SolarWinds was firewalled so it wasn’t allowed access to any random Internet address, and prevented the malicious code from contacting any command and control serve?

My concerns are two fold. While SolarWinds being negligent in securing its supply chain is obviously a disaster, the failure of customers to limit access to and by systems to allowed/required addresses and denied to all other addresses is a failure of a much more fundamental nature. This failure would indicate that many(most?) other systems would also be potentially available to engage in unauthorized communications with malicious actors.

Thus my second concern, that we are misidentifying the root cause that made this exploit possible, and are not focusing on the wider threat posed by inadequate, misapplied, or possibly non-existent access control policies. Supply chain compromises are a vector for malicious code insertion, but if those compromises require internet access to be exploited, and the systems compromised are at risk simply because their allowed communications weren’t constrained to only those required for system functionality, then that is the element of network security that is a larger problem.

If I let my dog roam free, is it the dog’s fault if it gets hit by a car? I have no way to know how secure the software I rely on is, but I can at least control who is allowed to talk to it. If that opportunity to implement gateway authority is not exercised then operators are not minimizing their exposure to risk.

I’m certain elements of my analysis are naive, I am not a security expert. And I still remain concerned that access control is not receiving as much attention for its contribution to the success of this exploit, and many others like it.

Etienne January 5, 2021 10:43 AM

Bottom line: FireEye must reveal all exploits it used to hack into computer systems, and alert the software companies as well as users.

There is no way, the U.S. Government can allow FireEye to win future contracts, and the company should be sued into bankruptcy.

Clive Robinson January 5, 2021 10:57 AM

@ Etienne,

Bottom line: FireEye must reveal all exploits it used to hack into computer systems

How do you figure that out?

Remember FireEye is effectively one of a very great many victims, they came forward as having been attacked which started this whole SolarWinds Orion hack.

So do you think also Microsoft should reveal all it’s code as well, because it’s another victim that has also come forward?

Etienne January 5, 2021 11:06 AM

Clive, FireEye certainly was a victim, but like the NSA was a victim, it’s hacking tools are now distributed worldwide.

This means that all computer systems are vulnerable unless we find out what was in all their hacking tools they surrendered to the enemy.

We have to look beyond the crime, and now prepare for the Pearl Harbor, wherever it may occur.

Those who left their doors unlocked, must never be trusted again. FireEye should be liquidated.

flasker January 5, 2021 11:55 AM

@Etienne

FireEye’s hacking tools relied on known vulnerabilities with existing patches and were used as part of the penetration testing services offered by FireEye that companies pay for. The NSA’s tools used unpatched vulnerabilities that had not been released to the vendors. I think that you are leaving out some important details about the two events you are comparing.

Clive Robinson January 5, 2021 1:38 PM

@ Etienne,

We have to look beyond the crime, and now prepare for the Pearl Harbor, wherever it may occur.

Those who left their doors unlocked, must never be trusted again. FireEye should be liquidated.

So as Microsoft clearly “left their doors unlocked”, and as a potential result with a way worse “Pearl Harbor” event to happen… then Microsoft “should be liquidated” under your reasoning yes?

You appear to have something personal against FireEye / Mandiant, care to share?

Tom January 5, 2021 1:56 PM

Can you really trust the intelligence services who are shouting “Russia”, when in fact they do not have more than guesses and clues that might also have been planted. (Maybe the work was mostly done between 9am and 5pm in Moscow, etc.) Now how can you know if the Chinese or North Koreans are not planting these well-known clues to point at another country? You can’t. It may have been Russia, but as long as there is no actual evidence, we should not believe the intelligence services.

No One January 5, 2021 2:06 PM

@David Glynn

Yes. I know for a fact that application servers that did not have firewall or web proxy access to the blind internet prevented the hack from doing anything. Also installing and running the application under a local only no-priv-ed account prevented the hack from doing many things. Also automatic updates turned off.

AlexT January 5, 2021 2:31 PM

Since the Snowden leaks we now that the NSA has tools that purportedly leave “traces” that will make some attacks / intrusions linked to a given country (aka Russia). We also know, as general paradigm, that attribution is always difficult. And we know that there is an ongoing pressure in the Anglo Saxon world to antagonise Russia, for whatever good it does to the world…

Not to say Russians didn’t do it – they might very well have. But if they had I’d be extremely surprised that the NSA “Early warning sensors” (ie penetration in Russian cybersphere and constant monitoring of traffic) did not trigger any warning. Could it be because… err… Russia didn’t do it ? And the line about legal prohibitions on the National Security Agency from engaging in domestic surveillance is laughable. Who cares ? Who enforces that ?!

On a larger perspective I am somewhat dismayed by Bruce record as of late, shifting from level headed, technical and factual blogging to parotting neoliberal ideas (if not propaganda), to the point of publishing oped-ed in such despicable publication as the Guardian. I don’t think he needs the money so there is something else at play here, which is troubling.

SpaceLifeForm January 5, 2021 3:18 PM

@ AlexT, Clive, Tom, flasker

Sometimes you must backtroll.

Looks like one bit. Maybe a second.

AlexT January 5, 2021 4:19 PM

It simply doesn’t matter. It happened, and it will happen again.
That is what matters.

Agreed. And lerning from it.

I am currently working with a team implementing a “zero trust” approach for the redesign of one of our major customers network. Very interesting and eye opening process ! There are lots of things you can do it turns out.

Tim bradshaw January 5, 2021 5:01 PM

@Clive Robinson

A few years ago (well, ten?) I had an argument with someone who was very keen that a system be code-signed. This was a Lisp system … with a compiler which could produce (native- but it doesn’t matter, even an interpreter would do) code from user input, which could do … anything. I failed to convince them that code signing was mot a magic bullet.

MSB January 5, 2021 5:30 PM

Imagine this sort of behaviour in ANY other industry today!
Cell phone batteries catch fire, and Samsung is facing lawsuits within days. Car tires blow-out on the highway, and Goodyear is facing lawsuits, and recalling millions of cars at their own expense.
Why are these standards not applied to software?
Solarwinds needs to become an example here. They need to be sued into oblivion and the directors need to be charged with [something–negligence?] and go to jail.
That is the ONLY way this sort of behaviour will EVER be resolved.
Right now, it is easy to externalize the cost of shoddy code. software so-called-engineers have no meaningful standards or certifications required to practice their trade.

Clive Robinson January 5, 2021 7:14 PM

@ Ken Hagler, AlexT, TimH, Tom,

I still don’t see anything in the article about evidence that Russia had anything to do with this.

I don’t see any evidence publicly even FireEye who are usually keen to jump which ever way the US Administration is pointing has been quite quiet on that.

What we have seen is one politico jumping up and down going, Russia, Russia, Russia and “dog whistling” which is not even remotely close to what I’d consider evidence.

Just remember the US Gov was even more certain it was North Korea with the olympics, when in fact it turned out that it was more probably Russia…

Clive Robinson January 5, 2021 7:39 PM

@ Tim bradshaw,

with a compiler which could produce (native- but it doesn’t matter, even an interpreter would do)

You should be surprised but probably won’t be by the number of software packages that have the equivalent of interpreters or escapes to other programs but at the same privilege etc.

It was bad news a quarter of a century or more ago, and it’s no less an issue today.

Code Signing as you note does nothing to save the end user from such nonsence.

SpaceLifeForm January 5, 2021 7:53 PM

Someone may have tipped their hand.

hXXs://www.politico.com/news/2021/01/05/dhs-cisa-company-data-solarwinds-455229

cybersecurity agency recently rejected requests … to share private companies’ confidential information

That's Interesting January 6, 2021 2:13 PM

Not sure whether it’s on one or another of two old emails, or on my name, but …. one comment of mine did get posted and the others that were put in moderation didn’t.

SpaceLifeForm January 6, 2021 5:58 PM

So, only about 30K DOJ email accounts.

hXXtps://www.forbes.com/sites/thomasbrewster/2021/01/06/doj-admits-microsoft-email-accounts-were-hit-in-solarwinds-attacks

Clive Robinson January 6, 2021 7:19 PM

@ SpaceLifeForm,

30k DoJ Email accounts, I wonder if they are wikileaks bound 😉

Clive Robinson January 7, 2021 5:12 AM

@ Tatütata,

OK, to tell the truth, I’m testing the censorbot to see if it is allergic to URLs…

The limit appears to be two.

But worse is the length limit and some kind of naughty word list. The length appears variable but short and “the place above” alone knows what the words are.

This blog is becoming “twitter” like, and getting a logical argument across nolonger possible except in silly little pieces.

Clive Robinson January 7, 2021 5:32 AM

@ Tatütata,

something called “JetBrains”, according to reports published Wednesday in various outlets

JetBrains is a developer tool developed in Eastern Europe, that apparently SolarWinds used.

Now if this is true, then what was claimed by the US Government earlier can not be true. That is their claim of “It’s Russia wot dunnit” would have no actual evidentiary basis when made…

Sorry this reply is in bits.

internet individual January 7, 2021 9:59 AM

I am seeing a huge uptick on my IDS from UK scanner traffic. Is it because NSA can’t scan our own domestic traffic that they need to outsource to UK? Does that mean MI6 had to hire a bunch more people to scan our traffic lol? Crazy world in which we live, you couldnt make the stuff up if you tried.

Security Sam January 7, 2021 5:39 PM

Some of the compromised software code
Was designed in an Eastern Europe node
And American sleuths are now examining
If such incursion latent evil forebode.

Clive Robinson January 7, 2021 7:57 PM

@ Internet Individual,

I am seeing a huge uptick on my IDS from UK scanner traffic.

That’s possibly due to the amount of traffic comming from other parts of the world through Europe and the UK.

Where the traffic orriginated is another matter.

With regards,

Is it because NSA can’t scan our own domestic traffic that they need to outsource to UK?

Unlike other aspects of “citizen surveillance” the NSA do not need to do that.

Remember there is a big difference between “domestic” as in within the land borders as defined by law and two ends of a cable.

It is highly probable that the IP level of your traffic is by no means the lowest level, it could be sitting on ATM or other entirely different network that just “bridges the traffic” from one place to another.

So you have no way to tell what the physical route of your traffic is. Thus if it goes out of the US mainland and back in again you have no idea –except by time– it might have been sent via Hawaii or some other place outside of US “domestic territory” thus is now legaly “open game” to the NSA.

JonKnowsNothing January 7, 2021 9:44 PM

@ Internet Individual, Clive

re: Incoming or Incoming

iirc(badly)
In the USA there are laws for “domestic surveillance” and “not domestic surveillance”. In order to obtain their desires and depending on who, what, where and why they want to collect a particular haystack, the NSA and others play some fun games with the courts and the FISA Court is prominent court they plan ball in.

They can reroute domestic travel thru another country and loop it back inside the USA. Since the loop back is from outside the borders, they can collect it as foreign data. Known loops used to run through Canada and back to the USA and from Miami out to points more southern and back again.

To get full interior data harvest they have to persuade FISC to allow it. FISC is very accommodating on this.

To sit on an international cable that is outside the USA, they can sit on it as long as they like. Any exit beach can be used. Just follow the coast lines and count the options.

Because we don’t really want to fully occupy a region, we hire the local teams to help out. It’s good PR to make them feel wanted, and gives them a cash incentive to cooperate. Regime Change happens if they do not.

The FISC dance only applies if you are inside the USA anything outside is open hunting.

Be mindful that everyone+dog sits on the international cables. There’s a fleet following the cable laying ships. Ocean cables get broken all the time and the cables need repairs. Taps are added for the same reason.

ht tps://www.theguardian.com/world/2020/dec/31/indonesian-fisher-finds-drone-submarine-on-possible-covert-mission
(url fractured to prevent autorun)

SpaceLifeForm January 8, 2021 3:32 AM

@ MSB, LSB, Clive

Are you guys Big Endian or not?

Which of you two are the most significant bit?

Fess up, or we will have to flip you.

Clive Robinson January 8, 2021 4:42 AM

@ SpaceLifeForm,

A “Chicken and egg” question…

Mind you they are both correct in there own points.

The real issue is software is sold not as a “good” with title but as a “service” with either a one time payment non transfereable “licence to use” or subscription but in either case no rights.

If this practice was stopped, and it could be fairly easily then most of the nonsense we currently have would fairly rapidly cease to exist.

However in effect FOSS could be killed off.

name.withheld.for.obvious.reasons January 8, 2021 7:00 AM

@JonKnowsNothing
On the routing or calls, the NSA after the 1980’s regular rerouted long haul links in and out of the country after the FISA court was formed by the recommendations of the Church committee findings. The caveat was that court orders weren’t necessary if the target on one side of the call was foreign in origin. By routing calls, any call could be made to have one end-point from a switched perspective appear to be foreign–hey no need to bother the courts. This became routine. At least that’s what I remember?

What is problematic is the congress passed HR4681 that allows for non-public communications (meaning private) to be collected. A free pass for all in .GOV to use is included in the bill, this was signed into law Dec 2014 along with the IAA during an omnibus fever dream-based destruction of the 4th. This coincided with the legal use of propaganda by the government on its citizens during the passing of that years NDAA. Way to go, democracy. In the USA, contempt for the foundational source of its own governmental authority is given the respect of ….

Free Julian Assange NOW!

Tatütata January 8, 2021 12:43 PM

On the routing or calls, the NSA after the 1980’s regular rerouted long haul links in and out of the country after the FISA court was formed by the recommendations of the Church committee findings.

How would you have done this in practice on the old analog network?

Doing a hair-loop on an oceanic cable or satellite circuit to Europe or Asia added an objectionable amount of noise and delay, in addition to being very costly, and the additional signaling and call setup time would have been obvious to even a moderately technically inclined user.

Mexico had a rather messy phone network (a motley assortment of Bell and CCITT standard equipment). Destinations were reachable by special area codes under the NANP (903, 905, etc., IIRC), before the +52 IDDD code was eventually implemented.

The Carribean (809) wasn’t very well served either.

That would leave Canada, which was integrated in the Bell System, and had decent circuits on microwave. So technically, it would have been feasible.

But long distance service was under the control of the Long Lines department or operating company, and routing couldn’t have been dependent on the calling number in the days of Step-by-step, panel, crossbar or even 1ESS. (Automatic Number Identification was a big complicated thing on the mechanical switches, but the calling number wasn’t transmitted between exchanges with pre-CCS systems).

You could reroute bulk sections of circuits (eg: you want to listen to Michel Corleone at 516 456 7890, so you you get the BOC to reroute all incoming and outgoing toll traffic from 212-456 through a foreign point), but someone could have noticed at the Canadian toll exchange, and you also still need the collaboration of AT&T or its BOC to patch their routing table without some sort of warrant.

And after the 1983 divesture of the Bell System things also got more complicated.

My conclusion is that I doubt that this could have actually occurred. Evidence gathered using such shenanigans would have been inadmissible.

But there has however been cozy arrangements between service on foreign transit and terminating calls…

JonKnowsNothing January 8, 2021 1:30 PM

@Tatütata @name.withheld.for.obvious.reasons @Clive

re: How was it done?

In some cases with mirrors

re: Getting someone to do it?

Not always easy but often easier than you might think. Black box Pen Registers installed by court order for n-days. N+1 days later, the government contract installer “forgets” to remove it. Happy Daze.

re: Admissible in Court

Not relevant because that is not the purpose. Parallel Construction provides any legal court documents needed.

re: Long Lines or Back Haul

Why would you even think they didn’t or don’t or aren’t…

Because a piece of paper says not to do that? There are many other pieces of paper, like NSL, that say You Will and they comply fast enough. Some have dual purpose offices for Team Work on prem. Keeps the paper work tidy.

re: Old Tech

In Clifford Stoll’s book there’s a particularly interesting section on having to set up the trace on manual switches in Germany.

ht tps://en.wikipedia.org/wiki/Clifford_Stoll
ht tps://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)

With the help of Tymnet and agents from various agencies, Stoll found that the intrusion was coming from West Germany via satellite. The Deutsche Bundespost (the West German post office) also had authority over the phone system there, and it traced the calls to a university in Bremen.

(url fractured to prevent autorun)

Clive Robinson January 8, 2021 10:44 PM

@ Little Endian, SpaceLifeForm,

Flipping is a drag whereas Barrel roll is a slider.

You two should head for the trees I hear Chris Wallace has one that can multiply your efforts.

Wayne Meriwether January 9, 2021 3:27 PM

With due respect, doesn’t the ultimate responsibility for my company’s IT security lie with me and my colleagues? It is so convenient to blame the vendor, but doesn’t this incident just reinforce the need to redouble testing and validation? Pick whomever you wish to blame, but IMHO if my company is impacted by such a breach it is the security officer’s fault, or it is executive leadership’s fault for not funding security testing adequately.

JonKnowsNothing January 9, 2021 4:13 PM

@Wayne Meriwether

re: doesn’t the ultimate responsibility for my company’s IT security lie with me and my colleagues?

Why would you accept liability for something you have no control over? Something you do not know exists and/or something you have no ability to change?

You can put yourself in the stocks for things you do know about and do nothing to mitigate providing you had the ability and choice to do so.

In another thread, a poster suggested “downsizing” as a fix-it for failure. You could resign or abdicate under that viewpoint.

If you downsize from N to 1, and all you have is 1 soldier to fight the battle, that’s not very likely to succeed.

If you up size from N to N*N and still change nothing, that doesn’t solve the problem either.

The responsibility lies with those that can fix it, should have fixed it and didn’t. The responsibility is with those that know there’s a flaw and ignore it because it suits their business plans. The responsibility is shared.

If consumers can only purchase faulty items, they have limited options; they have no ability to fix or change things themselves.

A fix that bricks the item, isn’t a fix either.

The Right To Repair is still not a right.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.