Dependency Confusion: Another Supply-Chain Vulnerability

Alex Birsan writes about being able to install malware into proprietary corporate software by naming the code files to be identical to internal corporate code files. From a ZDNet article:

Today, developers at small or large companies use package managers to download and import libraries that are then assembled together using build tools to create a final app.

This app can be offered to the company’s customers or can be used internally at the company as an employee tool.

But some of these apps can also contain proprietary or highly-sensitive code, depending on their nature. For these apps, companies will often use private libraries that they store inside a private (internal) package repository, hosted inside the company’s own network.

When apps are built, the company’s developers will mix these private libraries with public libraries downloaded from public package portals like npm, PyPI, NuGet, or others.

[…]

Researchers showed that if an attacker learns the names of private libraries used inside a company’s app-building process, they could register these names on public package repositories and upload public libraries that contain malicious code.

The “dependency confusion” attack takes place when developers build their apps inside enterprise environments, and their package manager prioritizes the (malicious) library hosted on the public repository instead of the internal library with the same name.

The research team said they put this discovery to the test by searching for situations where big tech firms accidentally leaked the names of various internal libraries and then registered those same libraries on package repositories like npm, RubyGems, and PyPI.

Using this method, researchers said they successfully loaded their (non-malicious) code inside apps used by 35 major tech firms, including the likes of Apple, Microsoft, PayPal, Shopify, Netflix, Yelp, Uber, and others.

Clever attack, and one that has netted him $130K in bug bounties.

More news articles.

Posted on February 23, 2021 at 6:18 AM20 Comments

Comments

Jeff February 23, 2021 9:47 AM

One of the advantages of go’s package management that uses explicit full path to the package, i.g. github.com/foo/bar.

Clive Robinson February 23, 2021 2:52 PM

@ Bruce, ALL,

Clever attack, and one that has netted him $130K in bug bounties.

Yes, but the vulnerability class that is behind it is ages old probably by six decades or more.

It’s an obvious consequence of,

1, The “Code is data” or “Data is code” comments you hear. Where infact both are just “information” held as a “bag of bits”[1].

2, “Bag of bits” files that are just infornation and can be treated as Code or Data.

3, File info / name having no real relationship to the file contents.

The most obvious to the eye examples of where people have tried to fix this are magic numbers at the begining of text and similar file contents, and file type extensions appended onto the files names. Both of which are so easy to fake, it’s hardly worth mentioning it, which is why the problem continues.

Likewise “signing” of a file will not realy solve this problem just make an attacker take a different attack route.

This problem might well be a “better mouse trap” problem and may not be solvable.

To any one who thinks they have a solution, I would suspect they are not “thinking hinky” enough, and almost certainly not like an attacker.

[1] For those not familiar with the “bag of bits” name, a “bag” is the most primitive of abstract aggregate data types. The only form it has is that it is a “set” of “things” in this case bits.

However to be of use in the case of a file, it has to have the concept of order of things as well. That is whilst many bits can have the same value, in a file you are interested,in the bit value at a given place in the file. It is in effect the same as the primitive tape in a Turing Engine.

It is upto the “user” of the bag of bits to define structure and meaning. Thus in the case of a “bag of bytes” not just the size of the bytes is defined but also which end is the most significant bit or start bit depending on the function of the bytes (numbers, characters, bit fields, etc).

https://www.d.umn.edu/~jallert/cs1/projects/TheBagADT.pdf

lurker February 23, 2021 4:11 PM

@Clive, Likewise “signing” of a file will not realy solve this problem just make an attacker take a different attack route.

Again displaying my slowness of wit: for years I have noted systems that religiously check the hash of downloaded files, but surely this is only to protect against transmission errors. If a bad guy can upload a fake file then he must also upload a [fake-file]hash.

Does having the file and its hash on the same server make this task any easier?

xcv February 23, 2021 10:06 PM

Another perennial supply-chain issue:

Executioners sanitized accounts of deaths in federal cases

A closer look at midazolam and pentobarbital

Whether it’s a federal or state execution, namely that of obtaining legal supplies of drugs for execution of inmates.

LOS ANGELES COUNTY SHERIFF’S DEPARTMENT
Los Angeles Deputy Says Colleagues are Part of Violent Gang

The allegations against the Compton deputies follow accusations of other gangs in the department — called the Spartans, Regulators, Grim Reapers and Banditos — that also share tattoos and a history of violence, the Times said.
A violent gang of Los Angeles County sheriff’s deputies who call themselves “The Executioners” control a patrol station in Compton through force, threats, work slowdowns and acts of revenge against those who speak out, a deputy alleges in a legal claim.

Mr C February 24, 2021 12:27 AM

@ Clive, ALL”
“This problem… may not be solvable.”

It appears you’re thinking of this problem so broadly I have trouble following. Taking the problem in narrow terms the solution is obvious.

This problem, defined narrowly as “build system grabs dependencies from the wrong repo” is readily solvable. Just eliminate the ambiguity. Like Jeff said, the build system should require unambiguous paths, meaning full urls for any resources that aren’t on the local machine.

The related, broader problem of “today’s clean v1.0 dependency becomes tomorrow’s poisoned v1.1 dependency” also has an obvious, if painful, solution. Clone a known clean version of the dependency to the local machine and point your build system to the local copy. Review commits fastidiously before updating the local copy. No one is going to like doing this, since it adds a huge labor cost to what they like to think of as a free product, but that’s the price to be paid if you want to be sure that free code is also safe code.

Two more general things that might have prevented this: (1) Don’t treat dependency names as if they’re secret; they’re not. (2) Don’t use javascript for anything important.

JonKnowsNothing February 24, 2021 1:42 AM

@Mr C @Clive @All

re: Eliminating Ambiguous Dependencies by fully qualified paths

Just eliminate the ambiguity. Like Jeff said, the build system should require unambiguous paths, meaning full urls for any resources that aren’t on the local machine.

I may have missed something but… what do you plan to do when the URL changes?

Lots of repositories re-org regularly. Re-sort, re-layer, re-tree and move servers. URLs are not permanent. Some URLs may get blocked by LEAs, Government rules or laws, TOS/EULA fights or just plain dueling mega corps. What worked today may not be in the same spot tomorrow. They are not permanent because someone has to pay for the name and if they don’t pay or get bought up, your path isn’t going to be where you thought it was.

It the Old DOS Days, we did fully qualified path names like:

  F:/Docs/My Docs/The Docs/That Docs/New Docs/FindMEaDoc

It didn’t work any better…

Clive Robinson February 24, 2021 2:52 AM

@ Mr C,

This problem, defined narrowly as “build system grabs dependencies from the wrong repo” is readily solvable. Just eliminate the ambiguity.

That’s a “moving the problem” reaction not a “Solving the problem” solution. It’s akin to playing “whack-a-mole”.

To do what you suggest requires things be set in concrete, but the network and the rest of the infrastructure are not for good reason.

So look at things in that light and you see the application gets told by the OS or Network that “Red is Green or Green is Red” and thus the application believes it because it has no way of telling.

Some how it is the “bag of bits” it’s self that has to prove to the application it is valid.

The obvious next step would be that the file is somehow encrypted and signed… But when loading multiple files of code from different places you have multiple signitures, and we know what fun can happen there with the problems with CA’s. Then there is the problem of how do you know the signiture you get is valid? Baking it in is likewise not a good idea especially when the latest thinking about PK certs is you have to change them once a year anyway…

I could go on but as I’ve said,

“To any one who thinks they have a solution, I would suspect they are not “thinking hinky” enough, and almost certainly not like an attacker.”

Clive Robinson February 24, 2021 3:05 AM

@ SpaceLifeForm, ALL,

Small fire. Just a flesh wound. Sound on.

Well atleast it looks like they had nice weather… Oh and no gas line got ruptured yet…

But a “Thank God” from the local Sheriff after the event is not a valid safety precaution, by any measure.

Oh and that black smoke is almost certainly full of carcinogens that are going to end up being breathed in or ending up in the local water course, plants and live stock etc. So there may well be a death toll down the line.

Clive Robinson February 24, 2021 3:48 AM

@ yosef,

As always such claims appear to be founded on the notion,

“Americans brilliant, every one else is stupid, therefore must be stolen US technology”

Ever heard of “coevolution” where two entirely different branches of spiecies respond to each other. The usual example is “flowers and bees” which is mutually benificial. However thorns and stings to deter predation might be a better example.

But evolution has other tricks, bays and birds independently developed flight in response to the environment.

Even the Wired article eventually all be it diminutively makes the point,

“Check Point’s findings aren’t the first time that Chinese hackers have reportedly repurposed an NSA hacking tool—or at least, an NSA hacking technique.”

But nobody has yet dared say,

“Hang on it’s a two way street”

Both the Chinese and Russians are quite advanced when it comes to software. Russia because it was behind in hardware did not have the luxury of just throwing code at a problem and assuming the users could just have more RAM and a faster CPU they learned how to live well within very restricted resources. The Chinese got to where they are via a different route. The point is they are both more than a match for the US and likewise the NSA.

There are two ways you can find a new attack,

1, Code analysis.
2, Random trials.

Both are actually probabalistic in nature. Which does not rule out two people throwing double six at the same time. Or for a lone player to throw six of them in a row.

The fact that China, Russia the US, or any other country has a zero day does not mean that the others can not also have found it, or even found it first.

So there is a very distinct probability that the NSA is stealing and using other peoples attacks.

I certainly would, as it would help with “deniability” of actions or “false flag” operations.

My father impressed on me four or five decades ago that,

“When there is trouble, the best place to be is somewhere else”

However if you think about it if you have skin in the game then,

“When there is trouble, the best person to be is someone else”.

It’s why you will see,

Attribution is Hard

Mentioned on this blog fairly repeatedly.

The most dangerous thing you can do when carrying out an investigation is “to make assumptions” which is by and large what a lot of these “It’s the Buttler wot dunit” claims are at the end of the day. Worse they are often made for political or patronage reasons, with the result the water is so muddy nobody can see even as far as the end of their nose, which is also convenient if you want to hide your activities.

Dave C February 24, 2021 7:28 AM

@Jeff:

One of the advantages of go’s package management that uses explicit full path to the package, i.g. github.com/foo/bar.

Further than that, as of Go 1.13 (Sept 2019) they by default use and check “an auditable checksum database” (here checksum==cryptographic checksum) for all dependencies. This is for reproducible builds as well as security.

For anyone interested, see sum.golang.org and the link there for “Secure the Public Go Module Ecosystem Proposal”.

The end result is that if I first add a dependency on, say github.com/foo/bar, then I get some exact version of that has been seen before and is permanently recorded by the public database (trust on first use) and thereafter, unless I manually make a change to the required version, anyone (inside or outside my organisation) that builds my package will, by default, use the exact same version of the dependency that I used (and hopefully have fully tested against). If someone puts a modified version on github then depending on how they do it (e.g. a man-in-the-middle substitution or a github account take over or whatever) various checks will fail or at absolute minimum (if I blindly ask the tool to “update” to the “latest” version) there will be a record of the bogus hash in the public database for later auditing by anyone. In practice, not perfect, but quite nice.

SpaceLifeForm February 24, 2021 5:58 PM

@ lurker

Does having the file and its hash on the same server make this task any easier?

I would ask SolarWinds.

MrC February 24, 2021 7:51 PM

@ JonKnowsNothing:

I may have missed something but… what do you plan to do when the URL changes?

The build fails. Then a human being has to sort out where the reorg moved it to and update the dependency list. “Failing hard/loud” is preferable to letting the build system make a guess about what to substitute for the missing files.

Clive Robinson February 24, 2021 10:56 PM

@ Mr C, JonKnowsNothing, lurker, SpaceLifeForm, ALL,

Then a human being has to sort out where the reorg moved it to and update the dependency list.

“Bang you are dead” or more politely QED.

That is a “Grade A1” exploitable vector via “social engineering” and other techniques.

As I indicated it’s a game of “Whack-o-Mole” and I suspect not one for which a realistic security solution exists, or is likely to exist any time soon.

But look at it another way,

1, All software has bugs.
2, We do not want buggy software.
3, All software needs patching.

Thus no matter what we do, a human has to go in there to make changes to existing code.

In essence, that’s the root of how SolarWinds got used.

SpaceLifeForm February 25, 2021 1:04 AM

@ MrC, Clive, JonKnowsNothing, lurker, ALL

When doing your software build, you want to FAILFAST.

You want to know there is a problem as soon as possible.

This is why you want the complete codebase on your build machine(s).

You do not want to encounter unexpected dependency failures.

You certainly do want want to rely upon pulling code on they fly over the internet. Why have a dependency on internet connection when you should not need it?

That is not the smart option. Pull the source. Manage your own repository. Then you can build offline.

FAILFAST.

SpaceLifeForm February 25, 2021 1:14 AM

@ MrC, Clive, JonKnowsNothing, lurker, ALL

FAILFAST

Worked really well before. 11 or 17 lines? Your call.

hx tps://arstechnica.com/information-technology/2016/03/rage-quit-coder-unpublished-17-lines-of-javascript-and-broke-the-internet/

ht xps://www.theregister.com/2016/03/23/npm_left_pad_chaos/

name.withheld.for.obvious.reasons February 25, 2021 1:25 AM

Today during the confirmation hearing for the Director of the CIA, William Burns, I couldn’t help but be struck by the inane and nearly pointless conversations with Senators about supply chain issues. How many years has this topic been beaten to death on this blog–over thousands. Everything from vendor and manufacturer issues and sourcing, design and integration, and implementation, process, and source controls beyond infinitum. It is as if I’d taking the wayback machine and Mr. Peabody was instructed to set the dial to 1982.

SpaceLifeForm February 25, 2021 2:06 AM

@ name.withheld.for.obvious.reasons

Got to get fresh blood in Congress. Too many dinosaurs that do not grok technology or supply chains.

My googlefu is failing me right now, but I definitely recall a congress-critter that dissed farmers, and said he would just go to the grocery store.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.