Schneier on Security

antibozo • August 16, 2007 5:31 PM

Brandioch Conner> The problem is that ANYTHING you do could alert the intruder.

Obviously. But taking the box off the net is a lot more likely to alert him than ls -lu, especially if just before that you do a bunch of nslookups and wgets from the compromised host.

In addition, there are other reasons not to disconnect the box from the network. For one thing, you may be unable to collect all the information you need about the intruder’s network connections before they time out. For another, if you have NFS mounts on the box, taking it off the net impairs your ability to collect system information since things start hanging.

The general rule is to be as unobtrusive as possible until you’ve sussed out the full extent of the compromise and are ready to contain, then contain all the compromised boxes simultaneously so you’re not playing whack-a-mole with the intruder. The major exception to this is if you have a web defacement–in that case, the intruder expects you to take the box off the net, and the PR harm usually warrants doing so as quickly as possible.

Brandioch Conner> Following that, you couldn’t even reboot a router. Or shut the box down to swap a power supply.

Indeed, if you have a mass compromise, those are risky activities, and even if the intruder doesn’t notice, you may lose important information about the compromise. But if you know you have a compromise, dealing with it is usually the highest priority, so you shouldn’t be swapping power supplies. If you don’t know about a compromise, you just go about your normal maintenance activity.

antibozo • August 17, 2007 3:53 AM

Pat Calahan> If someone cracks into my personal *NIX box, I’m going to unplug the thing from the network and the wall. Then I’m going to yank the hard drive out of the machine, plug it into another box, fsck it and examine it if and when I decide I feel like a good murder mystery.

I’m reluctant to go into specific tactics for a specific yet vaguely defined scenario for a number of reasons, but here are a few thoughts about that course of action:

1. I assume you don’t perform any remote admin work for clients from your personal *NIX box, because all your remote credentials may have been compromised in this scenario. Without finding out, minimally, how long ago it happened, you won’t even have an approximate timeline to examine for all of your remote activity on such remote hosts, nor will you have any clues about which credentials to treat with priority. It’s a good thing you don’t do that remote admin work, because you’d be doing a tremendous disservice to yourself and your clients by not doing due diligence to identify any compromise of their systems resulting from the compromise of yours.
2. You just threw away oodles of information from the /proc filesystem, cores of the intruder’s processes, in-core log information, and kernel data structures that would help you determine the scope of the situation you’re dealing with. At the very least, saving a copy of everything in /proc onto a USB stick, along with, if possible, an image of /dev/{,k}mem is de rigeur.
3. fsck modifies the filesystem. You’ll get more info if you can avoid using it. You don’t need to mount ext3 or Sun ufs filesystems; there are programs such as debugfs and fsdb that allow you to inspect these without mounting, which is safer for your analysis system anyway.

Pat Calahan> You can leave it up if you like, but I don’t believe overmuch in performing forensics on a live machine, since you already don’t know what it’s doing.

Well, I wouldn’t leave it up indefinitely, but I’d certainly take the time to identify everything I reasonably could about live network connections before yanking any network cables. And I wouldn’t crash it until I got everything useful I could out of the live system, and believe me, that’s actually quite a lot, because rootkits generally suck.

Pat Calahan> In most cases, I think a reasonable security plan would say, “Kill the box immediately, and anything it has been talking to”.

In most cases, it depends on the box, the function, and the type of compromise. Take a look at the NIST incident response documents (e.g. 800-61, 800-86) for some idea of the range of possibilities here. In most scenarios, a partial network containment would be initiated early on, but there are always exceptional circumstances, and every incident is unique.

Pat Calahan> If I don’t have an emergency response plan that considers that I may have to forcibly shut down a major control center (or have it shut down for me), for whatever the root cause, and I’m responsible for something as important as a city power grid, I’m going to quietly go work on my resume.

I think you erroneously assume here that the incident response team, or any computer security professional for that matter, ever had any input into the emergency response plan for the compromised system. Incident response teams are like firefighters; they’re called in when there’s a fire. They aren’t consulted when the facility was built or the fire detection and suppression system was installed; they’re just expected to deal with the fire when it happens, and in many cases the people who built the facility didn’t follow the fire code, if you will. Few organizations have an inspection regimen for software and systems that parallels the physical inspection that covers facilities, so it’s way worse for response teams than for firefighters. On the plus side, incident responders have much safer jobs… ;^)

antibozo> Meanwhile, if I lose my chance to find out what IPs the attacker was last working from by pulling the network too early, I’m throwing away valuable information I could use to identify other compromised hosts by inspecting firewall logs.
Pat Calahan> Someone who’s going to build a self destruct is probably going to do a good enough job of covering his/her tracks that you’re not going to get much out of the machine, forensically speaking.

I mean that once you pull the network cable you have limited time to collect the network information in all cases, not just those where a logic bomb is involved. Sorry that wasn’t clear.

And, as observed elsewhere, by pulling the cable you start the game clock, so you’d better be ready to play.

Pat Calahan> Since the box is untrusted, you can’t rely on the logs anyway

That doesn’t mean you ignore them.

Pat Calahan> (most rootkits I’ve seen turn off logging as the first matter of course anyway).

You’d be amazed at how often people screw up this simple step.

In practice, there aren’t any good rootkits. Suckit was fairly innovative at the time, but it has major flaws and it is actually extremely easy for a competent incident responder to detect and build a network scanner for. Yes, the sysadmin may be fooled by it. That’s why step 1 for the sysadmin is not “pull the network cable”; it’s “call the incident response team”. After all, a lot of times, the box that’s acting funny isn’t compromised at all.

Pat Calahan> And, if they’re not good enough to wipe tracks properly, you’ll get plenty of dirt off of examining the contents of the drive and the swap file while doing your examination using a known trusted OS platform.

I’m afraid you are vastly understating the volume of information that is lost when a system is crashed. Just consider what the core image of an intruder’s shell can be worth. Don’t count on your being able to reconstruct anything useful out of the page-sized confetti fragments strewn all over the swap device. And a lot of systems have enough core these days that they never even touch swap.

New systems are an issue with disk imaging as well. How many 300GB system disk images do you feel like scanning through and trying to keep in online storage? And now that sysadmins don’t know how to partition disks any more (“oh, I just made one giant root partition”) you have almost no help from the disk layout in narrowing your search. Believe me, you’re going to find out a lot more a helluva lot faster looking at the live system than you are trying to scrape through a massive disk, after you’ve pulled it and found another box to plug it into. Sorry, but the notion of crashing a box without a preliminary exam in anything other than the most extreme case is totally insane.

Nice chatting with you, Pat. :^)

antibozo • August 17, 2007 1:01 PM

Pat Calahan> I don’t do any adminning from any box that runs services.

Oh, good. And you don’t ssh into any box that runs services with X11 forwarding enabled either, right? ;^)

Pat Calahan> If a machine is compromised, saying “you should always do this [list]” vs “you should always do this other [list]” isn’t the argument you should be having.

I know it’s a lot of noise to slog through, but I think you’ll find that if you go back and read everything I’ve written above, you won’t find me saying you should always do anything in particular; rather you’ll see that my position is that there are always exceptional circumstances and that no single course of action is necessarily correct in every case. In other words, [always] think, and try to observe, before acting.

Pat Calahan> You’re arguing a predominantly forensic standpoint, Brandioch is arguing a predominantly preventative standpoint, but you both have good ideas that may be more appropriate given the context.

I think that’s not accurate–I’m arguing a containment-driven standpoint, not a forensic one. The overarching context here is a Linux compromise, which may thus involve a box used to conduct remote admin functions, thus containing the incident may involve a lot of other systems.

The first operational goal in a compromise is containment, and the first step toward that goal is not necessarily pulling the network cable on the box you know about. That would be a greedy algorithm approach, and, as I’m sure you know, with complex problems, greedy algorithms are not always the most efficient. In some cases it can lead to a nasty game of whack-a-mole. You’ll win in the end, one way or another, but if it takes a week instead of a day to contain, and a month instead of a week for the syadmins to recover (because they have to reinstall more systems), that’s a lot of time wasted that could have been spent on preventing the next compromise.

Pat Calahan> You may have the router logs, and already know what traffic has been coming from the box. You may know that the box was running a web service with a PHP backend and already have a pretty good idea how the box was hacked, even if you don’t know the specific details.

If you already have that kind of knowledge, indeed, you have a lot less to do on the compromised box before acting. Again, all I’m saying is, “take some time to understand the situation before acting”.

Pat Calahan> Using a log host helps a lot

Hear! Hear!

Pat Calahan> You may be in a race for time to prevent the box from attacking other things inside your DMZ, if the box has access to your management net.

Definitely–you’re always in a race for time. But until you investigate, you don’t know where your opponent is in that race. There’s no point running a leg of the race you’ve already lost. If the intruder already got onto your management net and has his sticky little paws on all your switches and servers, there’s really no point in unplugging the network cable of the one box–you may need to go for the jugular at that point and hit a kill switch on the whole computer room. And if you’re going to have to do that, you may not want to give the attacker any lead time by taking finger-in-the-dike measures beforehand.

A lot of intruders are slow and unfocused. Some of them are lousy typists, and are running five network scans in parallel with working on your box and a dozen others. There’s even a reasonable chance the intruder is asleep at the time the compromise is discovered. If you’re focused and swift, you have a decent chance of finding out everything you need to know while the intruder does nothing of significance, and as long as he is unaware you’re working on the problem, he has no reason to hurry.

I should also clarify, as I think others have misconstrued my comments, that I’m not generally advocating delaying action in order to observe the intruder’s activity to try to learn more about his behavior and whereabouts; rather I’m arguing that you may wish to delay action until you’ve had a chance to survey your own systems, starting with the compromised box you know about, to understand the full scope of your compromise, so you can perform an instantaneous containment, closing all the doors at the same time. Sometimes this is a good idea and sometimes it isn’t, and experience helps judgment, but don’t rule it out just because someone told you to always pull the network cable immediately on a compromised box.

As Bugs Bunny would say, “It’s time for a little stragety.”

antibozo • August 17, 2007 2:43 PM

xrey> This may take weeks.

With any decent organization, it’s on the order of two or three hours from discovery to analysis of the first box, and from there another hour to hit a dozen or so more with one sysadmin working on that part of the problem in coordination with an incident responder. But you can double those times if the sysadmin yoinked the network cable before calling, and triple them if he rebooted.

The majority of the initial time is the sysadmin contacting the incident response team, the team getting the admin up to speed on procedures, downloading tools, etc. In parallel with that is discussion with sysadmins and managers about the network architecture and determining current critical service placement and downtime tolerance so that as the scope is narrowed down, the containment procedure can be ready to go in short order.

It will never, ever, take weeks to contain. Assuming proper staffing, two days at the outside for a complex system that provides life-and-property functions hence has stringent downtime requirements, and that may involve partial containment at a much earlier mark. On most other scenarios, three to four hours to contain, on the short side if the vector is found quickly.

xrey> Is a system which you left on-line and manipulated after compromise detection now inadmissable as evidence?

Criminal cases are extremely rare, and the objective in a criminal investigation is not necessarily containment of the compromise; it’s establishment of sufficient evidence to conduct a prosecution. This may be inimical to containment and is really a whole issue unto itself. I suggest you take a look at NIST 800-86, “Guide to Integrating Forensic Techniques into Incident Response”, as a starting point, though, if you’re interested. You might want to compare the forensic process discussed there with the incident response life cycle discussed in NIST 800-61, “Computer Security Incident Handling Guide”.

xrey> if my sysadmin(s) are busy playing spy-vs-spy over Machine A, they could miss the fact that the user credentials stolen via a key-logger from Machine A have now just logged into Machines B, C, and D which have all just opened up outboard connections….

If your sysadmins are busy playing whack-a-mole logging into B, C, and D to check for illegitimate logins (which may have been wiped out of wtmp), they could miss the fact that the .bash_history file on A shows the attacker logging into E, F, and G, and the fact that the attacker logged into A from 192.168.1.1 and your router logs show connections to the ssh port on H, I, and J from that same IP, and the fact that the attacker initially got into A using a vulnerability in service S, and K, L, and M are all running that service as well. If you don’t have incident responders who can figure out those things from A rapidly, yes, a scattershot approach may be your only option, but rest assured that plenty of organizations have capable incident response teams.

xrey> If I decide to leave the machine connected, am I legally liable for proprietary information stolen after detection (hospital records, etc.)?

If you decide to disconnect the machine, are you legally liable for proprietary information stolen at an increased rate from other systems because you just told the intruder he has to finish his task as quickly as possible?

Really, if legal liability in the type of scenario you’re talking about is your concern, your best ass-covering strategy is to down every external connection at your perimeter and kill power on every box, damn the consequences.

antibozo • August 18, 2007 8:29 PM

supersaurus> tools question: there are tools, e.g. tripwire, you can run to find out what happened in the filesystem. leaving aside the shutdown/pull the plug or not argument, do you use things like that at some point in the investigation (yes, I know, it matters how, where and when you set up the database and how you do the checks)?

For the most part tools like tripwire are used for either detecting a compromise or trying to recover a system after containment without reinstallation (rarely advisable, but sometimes necessary). They are not as useful for figuring out what really happened; they tell you that a file has been modified, but not when.

What is more useful during initial investigation are tools that build a chronology of filesystem activity, e.g. TCT/sleuthkit mactime. This mode of investigation is not foolproof because it is possible for an intruder to tweak timestamps deliberately, but this is very rare in practice, at least nowadays. A filesystem chronology helps you correlate intrusion activity with logs, both on and off the system, and shell history files, and gives you a rough idea of the sequence of actions the intruder performed. It also gives you an approximate time at which the intrusion began, and may be useful for developing signatures for checking for intrusion on other hosts. For example, if the first thing the intruder did was use wget to pull down a toolkit, and no one else uses wget regularly, you can check the last access time of the wget binary on all your systems to build a list of suspect systems that should be initial candidates for further investigation and possible containment. Timestamps also help when dealing with password sniffers; the last access time of the sniffer log usually tells you the last time the intruder looked at it, and passwords logged before that time are thus more suspect than those logged later.

Chronology tools are only useful, however, if the timestamps are pristine at the time they are collected, which is why timestamp collection is something that should happen as soon after detection of a compromise as possible. Every time a user does an ls, executes a binary, accesses a config file, etc., last access timestamps get changed. Normal activity can quickly erode the access chronology from the filesystem, making investigation significantly more difficult. If the sysadmin tracks down a sniffer log and cats it before collecting last access times, evidence of what the intruder knows is destroyed. Timestamps may also be reset by backups, cron jobs (the updatedb process that builds the fast find database for the locate command, for example), patching, and especially a reboot. A system that has been rebooted after initial compromise is a lot harder to analyze.

Of course, the ability to build a useful chronology is severely hampered by unsynchronized clocks. It’s important to measure the clock skew on a system during initial investigation so that timestamps can be adjusted to correlate with the rest of the world, and it’s an important practice for all sysadmins to keep system clocks synchronized. NTP is your friend.

If you’re interested in this field, download sleuthkit and play with it. See if you can independently construct a timeline of recent activity on your box as an exercise.

http://www.sleuthkit.org/

antibozo • August 19, 2007 12:36 PM

Gopi> Does it really take that long to pull the plug, rip out the drive, image it, and put it back in and turn the machine back on, if you want to avoid tipping the person off?

Yes, in general, especially with modern disks. A 300GB disk takes 50 minutes to image at 100MB/s, which is a generous transfer rate. And killing the box means throwing away lots of volatile information and completely disrupting the environment both for the system and the intruder. In contrast, collecting file metadata on a similar-sized disk would typically only take 5-10 minutes and cause no disruption.

Furthermore, the incident handler is often working remotely from the compromised system, so getting a disk image to the handler for analysis may be constrained by network bandwidth, and talking the sysadmin through the necessary steps can be iffy.

Gopi> If imaging would take too long, I just thought of a cool piece of hardware somebody should build. It would do a man-in-the-middle on your IDE or SATA bus. It would do a block copy as fast as possible, while passing through read requests from the CPU. If there was a write request, it would read and store the original block first.

Satisfactory live disk imaging for non-evidentiary purposes can be accomplished with dd and netcat, and doesn’t require taking the system down. On LVM systems, snapshots can be used to maintain consistency. In cases where you have a hot-pluggable server with RAID 1, you can do an instantaneous image without killing the box by failing one side of the mirror and pulling the disk. These are sometimes useful tactics, but again, from an initial investigation standpoint, the priority is a good filesystem chronology, and it’s usually a lot faster and less resource-intensive to collect file metadata on the live box than work with an image.

I believe devices similar to what you describe exist, however.

Your other suggestions about sleep mode and firewire are interesting. As far as sleep mode, I don’t believe I’ve ever seen a Linux server that implements a hibernation mode, but I can see it being workable with Windows laptops. I don’t do much investigation on Windows, so maybe someone else can comment.