Backdoor Found in Codecov Bash Uploader

Developers have discovered a backdoor in the Codecov bash uploader. It’s been there for four months. We don’t know who put it there.

Codecov said the breach allowed the attackers to export information stored in its users’ continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure,” the company warned.

Codecov’s Bash Uploader is also used in several uploaders — Codecov-actions uploader for Github, the Codecov CircleCl Orb, and the Codecov Bitrise Step — and the company says these uploaders were also impacted by the breach.

According to Codecov, the altered version of the Bash Uploader script could potentially affect:

  • Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
  • Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
  • The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.

Add this to the long list of recent supply-chain attacks.

Posted on April 21, 2021 at 11:12 AM22 Comments

Comments

metaschima April 21, 2021 4:43 PM

This is extremely shameful. Bash is not like C, it is a lot more difficult to create obscure code that has a nefarious purpose. The fact that a line was added that calls curl and nobody inspected it more closely before approving it is just appalling. Shame on the devs for missing this.

Ismar April 21, 2021 4:43 PM

Ok,
One more of these articles where not enough attention is given to the root cause of the issue-namely how the code base was compromised in the first place. In other words, what allowed the attackers to place the curl line in the code base, and stop usage of the code repositories of this type until the problem is fixed.
given that the checksum itself was not compromised should be a big clue in the investigation.

Anders April 21, 2021 4:52 PM

@Ismar

Root cause : “In this attack, threat actors had gained Codecov’s credentials from their flawed Docker image”

hxxps://www.bleepingcomputer.com/news/security/hundreds-of-networks-reportedly-hacked-in-codecov-supply-chain-attack/

Ismar April 21, 2021 8:16 PM

@Anders- Thanks,
I was reading the other arstechnica article, but still even with this article the cause is jut mentioned in passing with developers blamed for not checking the checksums more often- I am a developer and can tell you people don’t have time to perform these checks all the time. So maybe this checksum check should be included (automated) as part of the standard dev repository workflow

SpaceLifeForm April 21, 2021 9:25 PM

@ Ismar, Anders

My hunch is that the problem was originally discovered because someone noticed the unexpected outbound traffic. Then started looking deeper. Searched for ip address in their development tree, found the script. Then proved it via the hash.

You can get an idea of how many different projects have been impacted here

https://grep.app/search?q=https%3A//codecov.io/bash

David Rudling April 22, 2021 2:43 AM

Related. An unusual source has been identified for some vulnerabilities in the Linux kernel

ht tps://www.osnews.com/story/133327/university-banned-from-contributing-to-linux-kernel-after-intentionally-submitting-vulnerable-code/

Denton Scratch April 22, 2021 9:27 AM

This impresses me not very much:

bash <(curl -s https://codecov.io/bash)

That seems to be Codecov’s recommended way of running the bash uploader: executing some piece of bash script from over the internet, on your machine, without first inspecting it.

https://docs.codecov.io/docs/about-the-codecov-bash-uploader

There is mention made of a hash on that page; but it’s way down below the fold, way below the bash command.

I went there to find out what Codecov and their bash uploader are meant to do; the moment I saw that, I stopped reading and ran away. This doesn’t seem to be an outfit that cares much about security. Why are people still doing this?

I have no idea what benefit I’m supposed to gain by uploading information from my machine to this company. It says that what it does is:

detect the environment
gather reports
upload them to Codecov

That’s awfully hand-wavy. What’s in these reports? Apparently it searches the filesystem for code-coverage reports from CI systems, and sends them to Codecov. Are these uploads encrypted on the wire? How can I be sure they are not also uploading SSH keys, correspondence with my lawyer about my divorce settlement, or my Contacts file? Well, that’s easy – download the bash script and inspect it. But then why are they encouraging me to execute it without inspecting it?

epic-irc'r April 22, 2021 12:04 PM

metaschima: yes, shame on the devs for permitting it, but also, people need to become more open-minded just to the very possibility that their bash code might have bugs. I’ve seen a bug report on bash code turn in to a verbal turf war knife fight over nothing more than ego, based on the idea that bash is easy so anyone can do it, so anyone’s shell code is just as good as anyone else’s, and that’s ignorance plain and simple. It is ridiculous, and it must be mocked, mercilessly and with crumpets.

1&1~=Umm April 22, 2021 12:35 PM

@ epic-irc’r

it must be mocked, mercilessly and with crumpets

Hmm or should I call you “Idle, Eric” and ask you about Python?

SpaceLifeForm April 23, 2021 6:45 PM

@ Denton Scratch

The ‘hack’ was to upload the users shell environment variables.

There can be a lot of UI there (ex:DB creds), but if your ssh key is there, you should probably not be allowed near a computer.

Weather April 24, 2021 1:19 AM

@slf
Real hackers have already pwned the device, the script kiddies take there anger out on 127.0.0.1 🙂

Weather April 24, 2021 5:37 PM

@slf
I quickly whios aprnic and 127/8 it say there’s no registered domain ? Is it a Tla

YASCA April 24, 2021 11:15 PM

Yet Another Supply Chain Attack

https://arstechnica.com/gadgets/2021/04/hackers-backdoor-corporate-password-manager-and-steal-customer-data/


As many as 29,000 users of the Passwordstate password manager downloaded a malicious update that extracted data from the app and sent it to an attacker-controlled server, the app maker told customers.

In an email, Passwordstate creator Click Studios told customers that bad actors compromised its upgrade mechanism and used it to install a malicious file on user computers. The file, named “moserware.secretsplitter.dll,” contained a legitimate copy of an app called SecretSplitter, along with malicious code named “Loader,”

1&1~=Umm April 25, 2021 1:29 AM

@YASCA

It sounds so much more “SolarWinds OMGish” if you pull out a couple of different quotes 😉

“Passwordstate is trusted by more than 29,000 Customers and 370,000 Security and IT Professionals around the world, with an install base spanning from the largest of enterprises, including many Fortune 500 companies, to the smallest of IT shops.”

And,

“The breach is especially concerning because Passwordstate is sold primarily to corporate customers who use the manager to store passwords for firewalls, VPNs, and other enterprise applications.”

@ALL

Anyone taking a “book/pool” on how long before someone in the US Gov says “Russia” or one of the other three?

The reality is that the problem, that once used to be just a case of,

1, Limitations of the average human.

Now has further problems,

2, Users need to use multiple services concurrently.

3, The services people use are now based on the Internet / Cloud.

4, Nearly all users are now thanks to COVID and ISP policies not tied to an IP address or verifiable location.

Back in the early 1980’s the ‘Human limitations with passwords’ had got to the point that even then it was ‘a very serious concern’… So hear we are getting on for a ‘working life’ later or 25 IT-Generations later where our technology is supposadly 30million times more powerful, we’ve not solved the problem we’ve only made it worse with another human failing ‘convenience’.

So two human failings, -both of which are getting worse with time- have through one or two other human failings -stupidity arising from pride&greed- rendered what for atleast one Western Nation is ‘critical infrastructure’ of overwhelming National Security importance practically, not just theoretically insecure…

People need to understand that the life of the password should have been over by the mid 1990’s at the absolute latest. But it’s still here on full life support…

A password is in effect a ‘token’ and it has the same down sides every bit as bad as the ‘One Time Pad’, then a few more on top. Any cryptographer can tell you why the OTP though beguiling simple and with the promise of ‘Perfect Secrecy’ is in reality a very bad idea to use. Why they do not say the same about passwords is I guess more a question of ‘Why people do not ask them?’.

Oh and do not think most of the ideas about ‘two factor’ usage is going to help. 2FA systems are just as insecure but in different ways, again through the same human failings.

Weather April 25, 2021 3:25 AM

@1&1~umm
Why is it abad idea to use Otp that password have the same flaw…no dangling carrots 😉

1&1~Umm April 25, 2021 1:52 PM

@Weather:

“Why is it a bad idea to use Otp that password have the same flaw…no dangling carrots”

If you think about the big disadvantages of the OTP and likewise the ordinary password and make a list for each, when you compare them you will find that they have quite abit in common.

The difference though is whilst we avoid the disadvantages of the OTP, we just ignore the same disadvantages in the password…

It takes only a moment to realise that historically a password is a secret that should only be used once.

That is passwords when sent between a user and a computer are only as secure as the link in between. If as used to be the case with serial terminals, there was,

  1. No information security
  2. No electrical/elrctronic security
  3. Effectively no physical security

In essence the link was little different to a ‘broadcast’.

So for a while the link security was purely physical with metal conduits, significant walls, and in some cases armed guards. However for various reasons both resource and security they were insufficient as all purely physical systems are. The only security they provide is ‘time delay’ and then only if someone ‘walks the line’ every little while.

Slightly later anti tamper alarms were added. But be they mechanical, electrical or electronic they were like purely physical security all they replaced was some of the ‘walking the line’. They were still very much dependent on the time delays of the physical systems to alow for other response systems to be initiated. Also none of them were realy secure, due to the principle of ‘what man can make man can fake’. That is if an alarm can be installed it can be in some way defeated silently by an attacker so no alarm is raised.

Whilst this is less of an issue with physical objects, information objects like passwords cost next to nothing to copy, and are them just as usefull as the original.

Thus the security needed to move from a reactive model of alarms, time delay and response, to a proactive model. So eventually ‘information system protection’ was implemented. Usually in the form of some kind of link encryption. However before the late 1980’s encryption and the attendent issues was either eye wateringly expensive or increadibly slow. Also realistically not all that secure for various reasons. So it took the time delay out from minutes/hours of physical/electric security out to weeks/months of ‘commercial level systems’ to months/years for ‘state level systems’. Hence the reason for changing passwords every week or month, which we still slavishly follow these days though few understand why.

The problem with naff old crypto is that one attack that would be very difficult these days was very much simpler back when “link crypto” was mainly simple streams based crypro and thus subjecy to an “attack in depth”. Due to the way things work, almost exactly the same way for logging in knowing what and when the plaintext is would enable a low grade stream cipher to be broken relatively quickly, thus the password recovered.

There are other asspects to consider but this post is long enough.

If you assume not unreasonably that link encryption is always going to be vulnerable eventually without several other changes being made. Then consider a true one time password, it can not be guessed and the only way to attack the system is in effect grab the link in a form of MITM attack such that the attacker gets the access from the user and hijacks the session in some way.

There are ways such session hijacking can be prevented these days but they are generally not used.

The other problem as RSA token users found. Most supposed One Time Passwords are not random, but made by a crypto algorithm. If the “seed” is aquired then it’s game over as anyone can then reproduce the passwords. To help prevent this there are certain “time sync” based protocols built in as well as login counters, however the more secure such systems are made the more fragile they become in the face of user errors, thus they appear unreliable.

So ‘security -v- convenience’ becomes a trade off with conveniance winning in most cases.

Things just get one heck of a lot more difficult when “Single Sign On” systems are used. Many issue “tokens” that get put in the users environment in some way. Due to the fact users might have three or four Web Based systems open at the same time from the same browser, the lack of issolation present in such sustems means that some applications would be able to get the tokens of other sessions due to poor internal seperation in the browser.

Likewise password managers with web interfaces are not a good idea.

The only real solutions are,

1, Replace password systems.
2, Some how remove human failings such as poor memory.

SpaceLifeForm April 30, 2021 4:35 PM

@ ALL

Check your logs.

https://www.bleepingcomputer.com/news/security/codecov-starts-notifying-customers-affected-by-supply-chain-attack/amp/

Known IPs In Scope:

The originating IPs used to modify the bash script itself:

79.135.72.34

The destination IPs where the data was transmitted to, from the compromised Bash Uploader.
These IPs were used in the curl call on line 525 of the compromised script:

178.62.86.114,
104.248.94.23

Other IP addresses identified in Codecov’s investigation, likely related to the threat actor and associated accounts:

185.211.156.78
91.194.227.*

Other IPs that may be related to this incident (not confirmed by Codecov):

5.189.73.*
218.92.0.247
122.228.19.79
106.107.253.89
185.71.67.56
45.146.164.164
118.24.150.193
37.203.243.207
185.27.192.99

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.