Comments

spaceman spiff • April 27, 2010 1:36 PM

When you dry your laundry in the open, everyone knows the color of your shorts… Apache should be used as a prime example of how to dry your laundry in the open (sic).

Morley • April 27, 2010 2:18 PM

It’s incredible how much can be put at risk with just one simple security bug in a non-essential system. Quite a talented attack, but fortunately, not very destructive.

Kudos to apache for how they stopped it, how they recovered, how they learned, but most of all, how they told it all to the world.

Carlo Graziani • April 27, 2010 2:32 PM

Detection on April 9th, abatement and full post-mortem report on April 13th. That’s pretty fast footwork, considering the scale of the problem, the usual challenges in analyzing and thoroughly understanding their system logs, the effort required to migrate and re-configure services with large user loads, the formulation of short-term patches and long-term fixes, and the effort to write that extremely detailed (and refreshingly honest) report. They obviously had an experienced team of high-quality admins working round-the-clock on this.

Curt Sampson • April 27, 2010 3:27 PM

In “what didn’t work”:

SSH passwords should not have been enabled for login over the Internet. Although the Infrastructure Team had attempted to configure the sshd daemon to disable password-based logins, having UsePAM yes set meant that password-based logins were still possible.

This has bugged me for years, and is, I as far as I can tell, a serious design defect in the whole idea PAM itself, which is just too complex. But the sshd design is also somewhat to blame: if it’s going to hand off authentication to a system that will allow password authentication, that should be made clear. For example, the PasswordAuthentication setting choices should perhaps be, “Yes,” “PAMOnly” or “No,” with ssh complaining and refusing to start if PAM is enabled but PasswordAuthentication is set to “No.”

Curt Sampson • April 27, 2010 3:31 PM

Additionally: using a different password on every web site is of course a good idea. Exercise for the reader: how does this interact with the idea of OpenID?

John F • April 27, 2010 3:36 PM

@morley: “Quite a talented attack, but fortunately, not very destructive.”

That remains to be seen. My memory is fuzzy, but there was an issue several years ago where the source control system for a Linux distribution was compromised, and the authorization logic in some kernel source code was changed from “if ($EUID == 0)” to “if ($EUID = 0)”, or whatever the language appropriate equivalent was. The net effect was to take a comparison statement and tweak it very gently into a comparison that always returned true, but which didn’t look wrong at a quick glance.

From what I remember, it took more than a year for that little nasty to be caught, and people were actively looking for changes that had been made to the source code repository during the timeframe around the hack.

peri • April 27, 2010 4:10 PM

@John F: “if ($EUID = 0)”

That assignment evaluates to 0 so the if never succeeds.

Mailman • April 27, 2010 4:52 PM

From the blog entry: “if you logged into the Apache JIRA instance between April 6th and April 9th, you should consider the password as compromised, because the attackers changed the login form to log them.”

Of course, if the compromise is that deep, people should also question whether the blog entry itself is legitimate, and if the recommendation that everybody change their passwords is not an attempt to gather them in the clear.

NobodySpecial • April 27, 2010 5:16 PM

how does this interact with the idea of OpenID?

I would have thought OpenID was a solution to this. Unless you logged in directly to your OpenID provider and their system was compromised, by having only a token rather than a username/password at each site aren’t you rather more secure?

My understanding is that you authenticate the site at the OpenID end rather than entering any secret at each site.

tkil • April 27, 2010 5:50 PM

@peri: “That assignment evaluates to 0 so the if never succeeds.”

But any code after the “if” will have $EUID == 0; unless that “if” is the last block in the program, you now have other code running with root privileges.

John Panzer • April 27, 2010 11:36 PM

If the site accepted OpenIDs, it would not store your password, so it could not be stolen from a database. You still could be vulnerable to phishing if they changed the login flow and faked your OpenID provider; but the URL would at least be wrong, providing a chance of a defense.

A successful attack on the identity provider would be very damaging. Less so than a compromise of a password used across all sites though, as it’s more recoverable. And you can upgrade just the identity provider to use e.g. two factor auth and get 2 factor for all relying sites. Most real people use the same password for many sites today, meaning that their passwords are as secure as the least secure site on the Internet.

Curt Sampson • April 27, 2010 11:55 PM

Right: OpenID does have that potential phishing attack. And it’s a bad one too because, even though there’s “the chance of a defense,” a compromise of your OpenID credentials compromises every OpenID-using site you use.

I’m not convinced that there aren’t other ways, besides that phishing attack, of compromising your OpenID credentials, either. XSS looks like it has some nice possibilities for compromising a user’s current session with an OpenID provider.

My main point is that here we have the exact same problem we had with PAM above: we’ve replaced a very simple and well-analyzed authentication system with one that’s much more complex and has not had anywhere near the level of analysis applied to it. I’m quite certain we’re going to see the same results: new compromises in the wild that weren’t obvious during the design stages. Whether OpenID will eventually become as secure as merely using a different password on each site is an open question.

Note also that, while we now have good tools for dealing with name/password site login (such as the FireFox password manager and XMarks’ password synchronization), we don’t have such tools for OpenID. (Though admittedly they may not be as necessary.)

Personally, I simply stick with regular logins whenever I can at the moment. (I use a different 12-16 character randomly-genenerated password on each site.) I understand the risk model well and Firefox makes it extremely easy to use. It’s going to take a fair amount of work on my part to know what I’d be getting into if I switched to OpenID.

Curt Sampson • April 28, 2010 12:00 AM

Looking at other solutions besides OpenID, does anybody know the state of using asymmetric authentication with web sites? Using an SSH key to authenticate to web sites just as I do for logins on other machines seems to me an excellent idea, since it’s something that has already been well-analyzed (for at least the parts that are not new related to the HTTP/web site of the protocol), and it’s something that I understand fairly well.

Randall • April 28, 2010 12:09 AM

Competent pros in good organizations aren’t shy about fessing up: they’ve proven their chops and their bosses are rational enough that there’s no incentive to whitewash what happened.

OpenBSD discloses a lot; they’ve established their reputation, and their stakeholders understand that bugs happen. Most software vendors don’t.

It doesn’t hurt that a good security pro (or programmer, or sysadmin) has usually analyzed the cause and figured out how to keep problems like it from happening again.

Nostromo • April 28, 2010 1:52 AM

Slicehost doesn’t come out of this well:

‘the attackers via a compromised Slicehost server … We notified Atlassian … and contacted SliceHost… Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly’

In fact Slicehost comes out of this stinking to high heaven.

Nick P • April 28, 2010 2:29 AM

@ Curt Sampson

For the most part, that’s unlikely. Regular websites tend to do what they expect users to understand. Users understand entering a username and password. A good chunk know how to do it at least half-right. Usability studies of crypto apps as straightforward as PGP showed many people didn’t intuitively know how to use it. So, I’d doubt companies would invest in the development effort of mutual, public-key-based, single-signon authentication for their sites if hardly anyone would use it. Even then, few clients would use it right and the servers would be running on high risk, untrustworthy OS’s anyway.

On the other hand, I would love encrypted connections by default and strong, public-key-based mutual authentication. Again, though, it mainly seems to happen in non-public-facing networks.

C • April 28, 2010 9:09 AM

@Eli: “and detected the next day.”

No, it was only in the subsequent discussion on LKML that it was realised that this was a back door (quoted from your first link).

Readers wanting to learn more about back doors may look at Underhanded C contest winners’ entries. If you lack time, just look at Emmanuel Colbus’ submission on http://underhanded.xcott.com/?page_id=16 which, (1/256) of the time, innocently dumps the encryption key into the ciphertext file.

After reading these entries, you may stop using C for security.

lost at sea • April 28, 2010 11:35 AM

Perhaps someone could address the ‘why?’. That is, why would anyone bother to do this hack? Thoughts that came to mind is that Apache is used on a number of web sites and maybe the hackers were looking for weaknesses to exploit in attacking ‘juicier’ sites. Then I see that Apache has a lot of products. Then I see that Apache is selling itself to Oracle. But still, could someone address the ‘what’s in it for the hacker’ issue? Thanks.

Curt Sampson • April 28, 2010 11:40 AM

@Nick P

I think it’s possible that, given the infrastructure we’d need to add for PK auth on the web, it could be just as easy to use. I agree that PGP’s user interface is not great (and take a look at GnuPG for a disaster that surely contributes to insecurity–even I have trouble using it well), but compare with typical SSH implementations, which come closer to traditional name/pw usage. (Explanations of key files are necessary, but not difficult, in my experience.)

Yes, I could be mislead here by my long-time familiarity with SSH and that I don’t know more than the basics of OpenID.

Nick P • April 28, 2010 2:05 PM

@ C

Good point. That’s why high assurance software has to have precise specifications of behavior and every piece of code is shown correspondent to that. To deal with malicious developers, EAL6-7 software generally uses strong layering to isolate types of behavior, formal methods to show requirements-design correspondence, several reviewers signing off on each code submission, and a variety of testing. This kind of process will catch most defects or inconsistencies, intentional or otherwise. Still potential for failure here. It’s actually why I like Ada subsets for coding: you can almost entirely be sure what the code is going to do and hence have an easier time proving it corresponds to requirements without unnecessary (read: evil) additions. I’ve been looking into OCaml recently, though, because it was successfully used to make a DO-178B-certified compiler and the Coq theorem prover supports code generation for Ocaml. Besides, the actions of the OCaml compiler are so much easier to follow than GCC compiling a C/C++ program. (Can anyone follow that? 😉

@ Curt Sampson

Well, it’s possible but would be hard to get going. Such a future seems to have one of two likely scenarios: a centralized PKI infrastructure like DOD’s that dictated trust or every web site or user using a decentralized scheme like PGP where each site/user is identified/trusted by a asymmetric key pair. The first scenario has some scary possibilities that have so far ensured no uptake. The second scenario would require widespread reengineering of browsers, sites, etc. to support the new model. One issue that cropped up in my own designs is that cellphones are used by more and more people, but asymmetric crypto runs like a snail on these things. ECC crypto solves that problem but creates another: such esoteric math means we may not know how secure/insecure it really is and spotting backdoors in implementations would be harder. I prefer RSA because it’s so mature, but we need more accelerators and true random number generators in hardware for this purpose.

VIA’s PadLock they put on their C7 and Eden processors is the right direction, whereas Intel is wasting time speeding up already-fast AES when we really need to speed up authentication. Right now, due to lack of hardware support, users must wait an extra 1-5 seconds (on broadband) to support mutually authenticated connections. The choice this mandates is: be secure and slow; get work more quickly with some added risk. You know what choice they make. Better hardware support and possibly lightweight primitives (think Salsa20 with Skein) will cause more users to accept secure links by default. The other problem is that we will need to really secure the cellphones or create an alternative, convenient device solely to store authentication secrets and/or mission critical data. For phones, kernels like OKL4 or INTEGRITY provide the most promise and can do the job today. However, strong isolation technology still isn’t seeing much deployment and phone OS’s are too easy to compromise. This means that right now, with a very mobile crowd, performance and security are both too poor for a good PKI except in the most high end devices.

OpenID isn’t likely to take off and I’m not sure I want it to, but that’s another issue. Microsoft recently purchased and published an identity management protocol, U-Prove, that will probably get more traction as they deploy it within Active Directory federated services, CardSpace, etc. I heard the crypto specs are open, but I haven’t read up on it yet. You might want to look into that because Microsoft’s support almost guarantees it will see significant, if not widespread, use.

proceng • April 28, 2010 9:54 PM

@perl:[That assignment evaluates to 0 so the if never succeeds.]
Wrong. In the *NIX world (and other right thinking environments), a return code of 0 indicates success. This is so that a negative rc is an error and a positive one is a warning.
Don’t be mislead by the “C-ism” where FALSE==0 and TRUE==!FALSE.

proceng • April 28, 2010 10:50 PM

@lost at sea:
You said: [Then I see that Apache is selling itself to Oracle.]
Please provide some more information on this. The only web entry I could find was
“https://blogs.apache.org/foundation/entry/the_apache_software_foundation_receives”
(the Apache Software Foundation blog dated APRIL FOOLS DAY)

lost at sea • April 29, 2010 3:32 PM

@proceng

Please inspect my moniker. Next, please someone answer the question–what’s in it for the hacker to hack Apache.org’s list of bugs?

undefined • May 2, 2010 12:54 AM

@Curt Sampson

PAM can be configured to only allow OPIE, thereby not allowing password-based authentication. the “problem” isn’t PAM, but in whoever (mis)configured PAM.

each service that uses PAM can have it’s own PAM configuration file (named after the service, eg /etc/pam.d/sshd). in that file simply remove the undesired authentication scheme (eg “auth required pam_unix.so”, “@include common-auth”) and replace it with the desired one (eg “auth required pam_opie.so”).

yes, PAM is not security pixie dust you simply sprinkle on, but a tool that you have to use properly for best results.

Curt Sampson • May 4, 2010 2:12 AM

@undefined:

Given the complexity of PAM, of course it’s going to be misconfigured on a regular basis. Any security system has to take into account that people make mistakes quite easily. The designers of PAM appear to have given a lot more thought to flexibility than simplicity.

This could be mitigated to some degree by having PAM off by default in the sshd configuration file, along with a warning noting that if you turn this on, you’re activating a large and complex system that may do things you don’t expect, including ignoring parts of your sshd configuration file. But most distributions just leave it on by default, leaving users open to exactly what happened to the Apache project.

Asif • May 25, 2010 5:28 AM

I am very impressed by the detail of attacks. And it clarify the theary i read. If apache can crashed like this, then what should i expect from the computer i used, cookies of google and others ?

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.