Microsoft Signing Key Stolen by Chinese

A bunch of networks, including US Government networks, have been hacked by the Chinese. The hackers used forged authentication tokens to access user email, using a stolen Microsoft Azure account consumer signing key. Congress wants answers. The phrase “negligent security practices” is being tossed about—and with good reason. Master signing keys are not supposed to be left around, waiting to be stolen.

Actually, two things went badly wrong here. The first is that Azure accepted an expired signing key, implying a vulnerability in whatever is supposed to check key validity. The second is that this key was supposed to remain in the the system’s Hardware Security Module—and not be in software. This implies a really serious breach of good security practice. The fact that Microsoft has not been forthcoming about the details of what happened tell me that the details are really bad.

I believe this all traces back to SolarWinds. In addition to Russia inserting malware into a SolarWinds update, China used a different SolarWinds vulnerability to break into networks. We know that Russia accessed Microsoft source code in that attack. I have heard from informed government officials that China used their SolarWinds vulnerability to break into Microsoft and access source code, including Azure’s.

I think we are grossly underestimating the long-term results of the SolarWinds attacks. That backdoored update was downloaded by over 14,000 networks worldwide. Organizations patched their networks, but not before Russia—and others—used the vulnerability to enter those networks. And once someone is in a network, it’s really hard to be sure that you’ve kicked them out.

Sophisticated threat actors are realizing that stealing source code of infrastructure providers, and then combing that code for vulnerabilities, is an excellent way to break into organizations who use those infrastructure providers. Attackers like Russia and China—and presumably the US as well—are prioritizing going after those providers.

News articles.

EDITED TO ADD: Commentary:

This is from Microsoft’s explanation. The China attackers “acquired an inactive MSA consumer signing key and used it to forge authentication tokens for Azure AD enterprise and MSA consumer to access OWA and Outlook.com. All MSA keys active prior to the incident—including the actor-acquired MSA signing key—have been invalidated. Azure AD keys were not impacted. Though the key was intended only for MSA accounts, a validation issue allowed this key to be trusted for signing Azure AD tokens. The actor was able to obtain new access tokens by presenting one previously issued from this API due to a design flaw. This flaw in the GetAccessTokenForResourceAPI has since been fixed to only accept tokens issued from Azure AD or MSA respectively. The actor used these tokens to retrieve mail messages from the OWA API.”

Tags: , , , , , ,

Posted on August 7, 2023 at 7:03 AM • 30 Comments

Comments

Winter • August 7, 2023 7:45 AM

I would file this under Single Point of Failure and cross-reference this to Not your Keys, Not your Data.

Jon • August 7, 2023 8:34 AM

In Open Source (and Open Cryptography) having the source code should be irrelevant.

Obscuring the source is just security by obfuscation. The only things that need to be secret are the keys.

Yeah. Security is hard. To make people do security correctly, make themselves and their bosses financially liable when they fail.

Thank you, J.

pd • August 7, 2023 9:57 AM

US Gov email is hosted by … Microsoft?

I’m choking from laughter.

How many security failures do Americans need to experience before they release public private partnerships just don’t work where security is critical?

How many years since Snowden?

Like with guns, Americans just cannot learn that there are fundamental flaws intrinsic in their country’s basic structural core.

Or, if they do, there’s nothing they can do about it due to a broken political system.

Anonymous Coward • August 7, 2023 10:06 AM

In Open Source (and Open Cryptography) having the source code should be irrelevant.

It actually cuts a level deeper than that. If the key was generated on the HSM, it should be impossible to export it in the clear, regardless of how much source code you have.

This either means that whatever HSM Microsoft was using was completely compromised, or (more likely) Microsoft’s security practices are so bad that they generated the key in software, imported it to the HSM, and also left a copy lying around somewhere

Crrraaig • August 7, 2023 10:15 AM

At this point, the negligent security practice is choosing Microsoft in the first place. Hard not to use them on the desktop, but for your server/infrastructure layer it is simply bad form to go with MS or any MS backed solution. We need to hold accountable those that choose Microsoft, not just Microsoft themselves.

Clive Robinson • August 7, 2023 11:03 AM

@ Bruce, ALL,

Re : One Ring issue.

“The phrase “negligent security practices” is being tossed about—and with good reason. Master signing keys are not supposed to be left around, waiting to be stolen.”

The problem is as always control of “The root of trust”, loose it and all goes south from that point onwards.

A “master key” system is just another failure prone hierarchical “signing system”. In effect a Tolkien “One Ring to Rule them all”.

We’ve known for over a decade and a half that “signing systems” are not just insuficiently robust they are “blow up in your face fragile”.

These days[1] I would argue,

“Any security system based on current ‘signing’ systems is ‘negligent security practice’.”

And it’s something security researchers should have been looking to solve since the 1980’s.

We have M of N key share systems, but they still generally end up with a single integer as the “Root of Trust” (which once disclosed…).

When you are talking about “secrecy” by encryption working on a message by message basis which is how encryption was used up untill a half century ago such fragility was desirable.

However “individual message secrecy” is now a tiny fraction of what “security” now covers. So we need to be thinking on designing systems that are much more robust. One part of which is getting rid of the “One Ring…” model.

Back when looking at nuclear weapons not only were there “Permissive Action Links”(PALs) that had to be set correctly the delivery systems were usuall “two key”.

But before that bank safes/vaults were designed to have two part combinations, where one part was known to the manager and the other part to the assistant manager. So both had to enter their combination for the door to become unlocked.

So the idea of “security” rather than “secrecy” systems has long been known about.

[1] Actually I’ve been arguing it for over a decade or so especially with “Code Signing” and others with regards “Certificate Authorities”(CAs). I can not remember off the top of my head just how short in days it’s recommended to make Web Site Certificates to expire these days, but it’s something like “A year, a month, and a day”.

AL • August 7, 2023 12:03 PM

I’d be worried if these hackers weren’t confined to China, Russia, Iran and North Korea. And maybe Venezuela, and Cuba.

We’d be in trouble if other countries started with this hacking business. But, since all bad actors are confined to just 6 countries, I’m not worried. 😉

Ted • August 7, 2023 2:14 PM

Anonymous

This is why I am never using Outlook at all, ever not once.

Wiz, a security firm, said they believe: “only Azure Active Directory applications that work with Microsoft’s OpenID v2.0 were affected.”

Unfortunately, this would include not just Outlook and Exchange Online…

… but also SharePoint, Teams, OneDrive, and applications that support “login with Microsoft” functionality. It could also include apps like Skype, Xbox.

Wiz offers a CLI command people can run to see which AAD applications in their environment might be affected.

https://www.wiz.io/blog/storm-0558-compromised-microsoft-key-enables-authentication-of-countless-micr

Charley_S • August 7, 2023 3:20 PM

Master signing keys are not supposed to be left around, waiting to be stolen. […] The second is that this key was supposed to remain in the the system’s Hardware Security Module—and not be in software. This implies a really serious breach of good security practice.

Does it? Normally, a private key stored outside a Hardware Security Module (HSM) wouldn’t be stolen, but might be copied by a criminal. As with the key to your front door, if it’s lost or stolen, you’ll probably quickly notice and re-key, whereas a secret copy could remain useful for quite a while.

It’s not clear from the stories whether “stolen” is accurate or is an incorrect colloquialism. But note that they do say it was an “inactive” key, which opens the possibility that it was literally stolen. Like, maybe someone walked out with the whole HSM that contained the key. Does that still count as “a really serious breach of good security practice”? Certainly it would be a failure of physical security, but if several employees were convinced to collude, it could be difficult to guard against—particularly given that it’s considered broadly acceptable to hire security guards via an independent company that pays them poorly, without requiring much other than a lack of documented criminality.

If, however, this was not a literal theft, I’d have to wonder whether the key provisioning and backup processes could be implicated. What’s the typical “good security practice” there? As Winter notes, one doesn’t want a single point of failure, which suggests a need for alternate and/or backup keys.

Also, I believe “AD” from the article headline stands for “Active Directory”, a Kerberos-based technology that, as far as I know, tends to be deployed such that a single key is ultimately and completely trusted (making “domain admin” the holy grail for any penetration tester). I know that multiple domains can somehow be joined, but does “best practice” require that? Does Microsoft’s software, or any alternate Kerberos implementation, easily allow one to require multi-admin confirmation before potentially dangerous operations such as privilege-changes?

Ismar • August 7, 2023 5:22 PM

Maybe opportunity for Microsoft to implement additional, user side, layer of protection for emails similar to what Protonmail already has? Or, would that be too inconvenient for our intelligence agencies then?

Bruce Schneier • August 7, 2023 5:30 PM

I, also, don’t use Outlook.

I am one of the few remaining Eudora users on the planet.

Bruce • August 7, 2023 5:42 PM

G’day Bruce.

I was a huge Eudora fan, but I thought they stopped updating it a decade (or nearly two) ago. Is that security through lack of utility? I would think using unmaintained software would be a risk??

Conrad • August 7, 2023 6:10 PM

I was a huge Eudora fan, but I thought they stopped updating it a decade (or nearly two) ago.

It was open-sourced in 2018 under a BSD-style license. According to Wikipedia, “In 2018, a ‘small team’ started working on repairing Eudora to update it for modern use, using crowd funding to facilitate their effort, branded as Hermes Mail.”

It looks like the last release was in 2020, and the mailing list has had 3 messages since 2021 (2 of which were people trying to unsubscribe). So, not very active, and possibly unmaintained. At least it can be audited. If it’s only communicating with (POP/IMAP/SMTP) servers Bruce controls, there’s little need to worry about the networking code’s security, which leaves MIME parsing as the main code to audit. And HTML, if it supports that, which is unclear (it lists “formatting and styles” as a feature, so I guess so, but maybe it’s optional?).

My Probation Officer & FBI Know Who I Am • August 7, 2023 6:54 PM

The US government is soooo huge that it really truly begs a question as to why is it that it does not have its OWN IT Infrastructure? Custom made hardware, in the USA-not made in Gina. Also proprietary software, made by the government agencies/employees, including own proprietary Operating System, and every single app that is needed because helllloooooo – it’s 2023 and it HURTS to see a handful of individuals, US Citizens, having everything made in Gina just because it made/makes them more money and EVERYTHING ELSE including national security is less important! DISGUSTING! Where there’s a will – there’s always a way – ALWAYS!

lurker • August 7, 2023 7:20 PM

@Charley_S

Once a key is outside the HSM it’s software, isn’t it? The difference between stealing and copying software is just something for lawyers to argue about.

And as for saying ActiveDirectory is Kerberos based, that’s a bit like saying a thumb-drive is mineral based. Is there any open-sourced protocol that MS has taken and implemented cleanly, sanely and safely?

Charley_S • August 7, 2023 9:57 PM

lurker,

The difference between stealing and copying software is just something for lawyers to argue about.

“Stealing” means taking a thing such that the rightful owner no longer has it. That’s not some minor technicality, and unless we have reason to believe it happened, it would be better to stick to factual language—for example, that some unauthorized party was able to use the private key. Which, for all I know, might still be in its HSM.

Lawyers don’t need to argue about this, because prosecutors know enough not to pursue charges like theft/larceny when the facts don’t support it. (On the other hand, if someone’s caught with the Microsoft HSM on their coffee table…)

And as for saying ActiveDirectory is Kerberos based, that’s a bit like saying a thumb-drive is mineral based.

I don’t know how to interpret this statement. A Kerberos client such as “kinit” can interoperate with an Active Directory server, so it seems reasonable to suspect their trust and data models are similar. And, perhaps, that Microsoft implemented it “sanely”. (Ignoring the question of whether the centralized trust model from 1989 is appropriate today, because MS wouldn’t have been implementing an open-source protocol if they’d unilaterally changed that.)

starbuck • August 8, 2023 3:52 AM

From my modest understanding of cryptographic systems, the key would have been stored in an HSM, configured as non-exportable, and there would be no copy of it anywhere else but in another HSM or in encrypted form.

If what I assume is true, I only see two options:
1. Microsoft failed to a level unimaginable by either:
– allowing a copy of the key out of an HSM,
– using an HSM that allows non-exportable material to be reverted to exportable.

  1. An HSM had a vulnerability and was directly compromised, without the need for a valid account. This would probably bring us to the question of network isolation but we would have a greater issue at hand: one of the HSMs currently used by Microsoft in Azure can be attacked remotely and its keys extracted.

In both cases, I think the situation is urgent and users need to be informed on what is really at stake.

Clive Robinson • August 8, 2023 5:26 AM

@ Charley_S, lurker, ALL,

‘“Stealing” means taking a thing such that the rightful owner no longer has it.’

Err no, it’s rather more than that.

It is,

“Denying the owner the rights and privileges pertaining to ownership.”

If I stand by your nice shiny Italian Supercar and pick up a hot blond because I give her the impression I not you own it, than that is technically theft.

I photograph through your window your realy stylish home, then claim I designed it to get someone else to give me work that’s also theft.

If a lawyer can establish a right of ownership, then it can be traded upon. You denying the alledged owner the bebifit of that right is stealing from them.

In certain parts of the US you will find that somebody you have no idea exists owns the rights to the water that falls on your roof and land. You “collect rain water” and you will face criminal sanctions.

PaulBart • August 8, 2023 10:39 AM

“Stolen” lol, good story bruh!

Global companies = global masters.
NSA wants a backdoor to spy on their chatt..citizens, Mossad wants a backdoor for the same, as well as spying on Iran, MSS wants a backdoor, for who knows what purposes…

When citizens realize all their governments are evil, regardless of the spoils, can a path forward be found.

Winter • August 8, 2023 1:10 PM

@PaulBart

When citizens realize all their governments are evil, regardless of the spoils, can a path forward be found.

So, the path forward leads to no government? Where can we find this people that is happy with no government?

Anon • August 8, 2023 11:55 PM

Is there any significance in using an INACTIVE key? Is there a vulnerability with checking inactive keys that allowed it to be extracted/copied/used to create tokens?

vas pup • August 9, 2023 4:19 PM

How safe is my data after a hack or leak?
https://www.bbc.com/news/technology-66451970

“If you believe your password to an account may have been compromised, it makes sense to change it.

But you should be mindful not to respond to any emails recommending this, as they could be attempting to scam you – instead you should visit the website as normal, and change
your password there.

This is also why it is important to have different passwords for different accounts.

By always keeping different log-in credentials, a future hack is less likely to affect you seriously as the hackers will not be able to use your data beyond accessing a single service you used.

Recent research from Barclays released on Wednesday shows that 87% of all scams take place on tech platforms, such as dating apps, social media and online marketplaces.

“Our data shows that tech platforms – particularly social media – are now the source of almost all scams. However, there is no current legislative or regulatory framework obliging the tech sector to support the prevention of these crimes, as there rightly is for banks.”

But there are a few simple steps you can take to stay safe online, which can help protect your data – and your wallet.

These include using a strong password for each site you visit, avoiding emails from people you do not know, and being careful when visiting unfamiliar websites.”

Lee Neubecker • August 10, 2023 3:47 PM

Bruce,
I appreciate your post.

There is a real problem within Microsoft that remains a clear indicator of internal compromise. In order to know if an Operating system has been compromised, an understanding of what the valid Software Signing keys should be verified. Hardware based APTs need a rogue trusted certificate to exist in the operating system in order for AV and other security software to be disabled, while allowing the implant to run silently.

In seeking to understand if an APT is active in a Microsoft OS, you must first verify what “Good” should look like. That answer should begin with verifying Microsoft.com’s PKI Audit documents. My search for these revealed two different documents with the same Docusign Envelope ID. My testing as well as that performed by another Computer Forensics expert I collaborated with indicated that every time a new version of a valid Docusigned document gets created, a completely different Docusign envelope ID is generated.

The latest PKI Audit allegedly performed by BDO appears to be signed by the Microsoft Crypto Lead, Raza Syed head of Azure. Other Audit reports by BDO I found were digitally signed by the auditor from BDO, which is more appropriate and provides clear attestation and accountability.

There are two version of the latest June 2023 Microsoft.com PKI audit report. They both have the same Docusign Envelope ID which indicates one or both are fabrications.
Google search for “DocuSign Envelope ID: D297A988-9557-42AC-8346-913DD1BACD8E”

https://www.microsoft.com/pkiops/docs/Content/seals/microsoft%20pki%20acs%20wtca%20indp%20acct%20opinion%20and%20mgmt%20assertion%20june%202023%20-%20final.pdf 12 pages

https://www.microsoft.com/pkiops/docs/Content/seals/microsoft%20pki%20wtbr%20indp%20acct%20opinion%20and%20mgmt%20assertion%20june%202023%20-%20final.pdf 14 Pages

One or both of these are edited pdfs and are not authentically signed docusign PKI audit reports.

BDO should be signing all pages of their “Independent” PKI audit report for Microsoft.com, not internal executives.

Raza Syed from Microsoft should be question about why he edited one or both of these and which Digital signatures are the authentic CAs. The same is true for Chuck Chan who used to be listed on older BDO PKI audit reports for Microsoft.

If there are conflicting audit reports bearing the same Docusign envelope ID, an investigation including asking Docusign whether any of these were authentically digitally signed should take place.

If the world doesn’t know which version is authentic, the world will remain cyber insecure.
https://www.microsoft.com/pkiops/docs/Content/seals/microsoft%20pki%20acs%20wtca%20indp%20acct%20opinion%20and%20mgmt%20assertion%20june%202023%20-%20final.pdf 12 pages

https://www.microsoft.com/pkiops/docs/Content/seals/microsoft%20pki%20wtbr%20indp%20acct%20opinion%20and%20mgmt%20assertion%20june%202023%20-%20final.pdf 14 Pages

One or both of these are edited pdfs and are not authentically signed docusign audit reports.

BDO should be signing all pages of their “Independent” PKI audit report for Microsoft.com.
Raza Syed from Microsoft should be question about why he edited one or both of these and which Digital signatures are the authentic CAs. The same is true for Chuck Chan who used to be listed on older BDO PKI audit reports for Microsoft.

If there are conflicting audit reports bearing the same Docusign envelope ID, an investigation including asking Docusign whether any of these were authentically digitally signed should take place.

If the world doesn’t know which version is authentic, the world will remain cyber insecure.

Microsoft and BDO need to be held accountable for making it perfectly clear which Certificate Authorities are authentic.

Clive Robinson • August 10, 2023 4:40 PM

@ Lee Neubecker,

Re : Nasties in the wood pile.

“Raza Syed from Microsoft should be question about why he edited one or both of these and which Digital signatures are the authentic CAs.”

Ouch…

As they used to say,

“Them’s fighting words!”

These days they get a side kick to send a personalized “notice before action” or similar…

A friend very recently found out MicroSoft UK’s cloud service is a compleat and utter mess and should be avoided. They failed to respond to legal warnings, and it got within a hairs breadth of the high court.

However the head person does respond to having notice served personally at her home address, informing her of her personal liability for losses…

She was daft enough to have left it easily tracable on the Internet… I found it in about ten minutes.

Hans-Joachim Knobloch • August 11, 2023 5:08 AM

@ Lee Neubecker,

The respective scope of the two audit reports that you linked pertains to different PKI hierarchies.

The first one is for the “Microsoft Identity Verification Root Certificate Authority 2020” and its associated intermediate and timestamping CAs,
the second for the “Microsoft ECC Root Certificate Authority 2017” and “Microsoft RSA Root Certificate Authority 2017 ” and their associated issuing CAs.

As a side note, none of these PKIs seem to have any direct connection to the compromised authentication token signing key (except, of course, the company that is supposed to take care of them).

HK • August 11, 2023 2:24 PM

I would like to know, if the attackers really stole the signing key, or if they were just able to access the signing function (e.g of the respective HSM). This is not the same. But both option would allow the attacker to sign things. However, the nature of the attack would be different.

a fan • August 13, 2023 6:00 PM

According to MS blog the certificate was revoked. If the keys were store in a HSM, MS would have stated something like: whilst stored on an HSM we decided out of precaution to revoke the affected keys.

One of the other issues, us around the key or in this instance signed token transparency. Not keeping details of what you sign is particularly dangerous.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.