SolarWinds Detected Six Months Earlier

New reporting from Wired reveals that the Department of Justice detected the SolarWinds attack six months before Mandiant detected it in December 2020, but didn’t realize what it detected—and so ignored it.

WIRED can now confirm that the operation was actually discovered by the DOJ six months earlier, in late May 2020­—but the scale and significance of the breach wasn’t immediately apparent. Suspicions were triggered when the department detected unusual traffic emanating from one of its servers that was running a trial version of the Orion software suite made by SolarWinds, according to sources familiar with the incident. The software, used by system administrators to manage and configure networks, was communicating externally with an unfamiliar system on the internet. The DOJ asked the security firm Mandiant to help determine whether the server had been hacked. It also engaged Microsoft, though it’s not clear why the software maker was also brought onto the investigation.

[…]

Investigators suspected the hackers had breached the DOJ server directly, possibly by exploiting a vulnerability in the Orion software. They reached out to SolarWinds to assist with the inquiry, but the company’s engineers were unable to find a vulnerability in their code. In July 2020, with the mystery still unresolved, communication between investigators and SolarWinds stopped. A month later, the DOJ purchased the Orion system, suggesting that the department was satisfied that there was no further threat posed by the Orion suite, the sources say.

EDITED TO ADD (5/4): More details about the SolarWinds attack from Wired.com.

Posted on May 3, 2023 at 6:13 AM10 Comments

Comments

iAPX May 3, 2023 6:37 AM

“Unusual traffic” is suspect traffic, that’s why traffic is monitored and everything “unusual” is logged to be audited if not immediately launching an alarm!

If they couldn’t have a good network hygiene when evaluating a new solution, there are few chances they do it for production systems where it’s more complex with a lot more traffic.

There is something really weird on this story.
Including “the company’s engineers were unable to find a vulnerability in their code”, naturally they won’t, you didn’t ask people that created code with a security hole to find it: with external help to identify it and reproduce it, they could fix it.

You don’t do QA by code developers, but by QA people, you don’t search a flaw by code developers, you use a hacker for that matter.

Archer May 3, 2023 8:41 AM

A month later, the DOJ purchased the Orion system, suggesting that the department was satisfied that there was no further threat posed by the Orion suite, the sources say.

Or — and you are free to call me cynical — the DoJ decided that purchasing the system would allow them to further explore how they might use this vulnerability for their own purposes before it was patched.

Wannabe techguy May 3, 2023 9:25 AM

@Archer
OK you’re cynical. So am I. But, you’re also most likely correct.

Mat May 3, 2023 2:14 PM

DoJ should have hired one of the Bruce’s audience here and this whole Solarwinds catastrophe would have put to rest at that time of DoJ evaluation. Better yet, SolarWinds itself should have hired someone here than some MBA who has no idea about Security Engineering.

With the software development jobs already outsourced to eastern part of the world long ago (hence the executives in the US has no clue what is in their own products!) and dummies in the US running the CISO/Security show who has no real knowledge about Cybersecurity, there is going to be even more critical exploits in coming years potentially life threatening.

Learn the expertise from pros like Israel, Russia, China and hire them in the US to to be really cybersecure. Of course, be vigilant on these folks as well!

iAPX May 3, 2023 2:29 PM

@Mat

A closed bug bounty program, including worthwhile prizes, on an isolated system should have also done the job.
With prizes up to 6 figures, you will be noticed and you will attract incredibly talented people!

And as always, an open bug bounty program is a great investment on security, given you value it and the prizes are accordingly set: any researcher will be eager win a prize instead selling a security hole to anyone.

Andrew May 3, 2023 7:30 PM

Misunderstandings in comments here that SolarWinds software flaws (vulnerabilities) were exploited. It was instead the system of distributing updates of their software to customers was attacked out of band. Not a type of bug for a bounty program – this is IT and product management intrusion.

ResearcherZero May 4, 2023 12:10 AM

@Andrew

May get some good aurora
‘https://www.swpc.noaa.gov/news/r1-r2-minor-moderate-events-3-may-2023

and also a meteor shower, early in the morning…

But seriously now…

As you were saying

A supply chain attack intrudes into the development environment.

In this case they swapped a file (SolarWinds.Orion.Core.BusinessLayer.dll). The update was then compiled with the modified file. It was carefully done so that it would not produce any alert, by changing flags to suppress warnings.

This meant that no one noticed that the update had been altered. It was only later when malicious traffic was detected that any initial investigation took place, but it was not suspected that it was the update to Orion at that time…

“At the time of the investigation, Volexity deduced that the likely infection was the result of the SolarWinds box on the target network; however, it was not fully understood exactly how the breach occurred (i.e., whether there was some unknown exploit in play, or other means of access), therefore Volexity was not in a position to report the circumstances surrounding the breach to SolarWinds. The machines involved in this incident had been rebooted several times prior to Volexity’s involvement in incident response efforts, meaning that a great deal of evidence that would have been in volatile memory had been lost.”
‘https://www.volexity.com/blog/2020/12/14/dark-halo-leverages-solarwinds-compromise-to-breach-organizations/

Mandiant detected the backdoor inserted in the update a few months later…

‘https://www.mandiant.com/resources/blog/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor

The following article has a pretty simple explanation for anyone who does not understand technical terms.

‘https://www.npr.org/2021/04/16/985439655/a-worst-nightmare-cyberattack-the-untold-story-of-the-solarwinds-hack

ResearcherZero May 4, 2023 12:30 AM

Processing and analyzing all the traffic on a network is not as easy as it sounds. Different protocols have different limitations.

Orion is network monitoring software, (like WireShark for example). Orion allows you to monitor all your firewalls and other systems on the network, including printers and anything else connected in the building. You can view the traffic for all those systems, including DNS and other communications.

There are video demonstrations of Orion and other management software on the web.

ResearcherZero May 8, 2023 3:07 AM

Putin stressed: “The most serious attention should be paid to information security.”

“President Donald Trump only acknowledged the hacking on Saturday almost a week after it surfaced, downplaying its importance and questioning whether the Russians were to blame. Russian state media gleefully seized on President Donald Trump’s remarks playing down the role of Russia in the hacking.”

‘https://www.dailymail.co.uk/news/article-9072833/Putin-celebrates-100th-anniversary-Russian-spy-agency-major-cyberattack.html

Trump’s comments caught the White House off guard as they attempted to square the president’s comments with Secretary of State Mike Pompeo’s remarks a day earlier saying that Russia was “pretty clearly” behind the hack, according to two officials with knowledge of the situation.

White House officials had drafted a statement assigning blame to Russia for the attack and were preparing to release it Friday afternoon but were told to stand down.
‘https://www.nbcnews.com/politics/white-house/trump-downplays-russia-hack-first-comments-massive-breach-implicates-china-n1251813

Golden SAML is a technique that allows attackers, once they have privileged access to the victim’s network, to impersonate almost any identity in the organization and acquire any type of privilege across almost all services of the organization (this depends on what services in the organization use SAML as their authentication protocol).
‘https://www.cyberark.com/resources/threat-research-blog/golden-saml-newly-discovered-attack-technique-forges-authentication-to-cloud-apps

Steve May 16, 2023 7:40 PM

@iAPX

Doing nothing and claiming to find no vulnerability got the contract sold, which is the only job that vendors like this actually care about. The job is selling promises and taking risks that they will never have to pay out on.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.