More Lynn/Cisco Information

There’s some new information on last week’s Lynn/Cisco/ISS story: Mike Lynn gave an interesting interview to Wired. Here’s some news about the FBI’s investigation. And here’s a video of Cisco/ISS ripping pages out of the BlackHat conference proceedings.

Someone is setting up a legal defense fund for Lynn. Send donations via PayPal to Abaddon@IO.com. (Does anyone know the URL?) According to BoingBoing, donations not used to defend Lynn will be donated to the EFF.

Copies of Lynn’s talk have popped up on the Internet, but some have been removed due to legal cease-and-desist letters from ISS attorneys, like this one. Currently, Lynn’s slides are here, here, here, here, here, here, here, here, here, here, here, here, here, here, and here. (The list is from BoingBoing.) Note that the presentation above is not the same as the one Lynn gave at BlackHat. The presentation at BlackHat didn’t have the ISS logo at the bottom, as the one on the Internet does. Also, the critical code components were blacked out. (Photographs of Lynn’s actual presentation slides were available here, but have been removed due to legal threats from ISS.)

There have been a bunch of commentary and analyses on the whole story. Business Week completely missed the point. Larry Seltzer at eWeek is more balanced.

Hackers are working overtime to reconstruct Lynn’s attack and write an exploit. This, of course, means that we’re in much more danger of there being a worm that makes use of this vulnerability.

The sad thing is that we could have avoided this. If Cisco and ISS had simply let Lynn present his work, it would have been just another obscure presentation amongst the sea of obscure presentations that is BlackHat. By attempting to muzzle Lynn, the two companies ensured that 1) the vulnerability was the biggest story of the conference, and 2) some group of hackers would turn the vulnerability into exploit code just to get back at them.

EDITED TO ADD: Jennifer Granick is Lynn’s attorney, and she has blogged about what happened at BlackHat and DefCon. And photographs of the slides Lynn actually used for his talk are here (for now, at least). Is it just me, or does it seem like ISS is pursuing this out of malice? With Cisco I think it was simple stupidity, but I think it’s malice with ISS.

EDITED TO ADD: I don’t agree with Irs Winkler’s comments, either.

EDITED TO ADD: ISS defends itself.

EDITED TO ADD: More commentary.

EDITED TO ADD: Nice rebuttal to Winkler’s essay.

Tags: , , , , , , , ,

Posted on August 3, 2005 at 1:31 PM • 28 Comments

Comments

Drew Miller • August 3, 2005 2:20 PM

I couldn’t agree more with you on your points. Any update/info on the “Abaddon@IO.com” fund would be appreciated.

Glauber Ribeiro • August 3, 2005 2:32 PM

After i read Lynn’s presentation, the impression i had was that if there is any vulnerability in Cisco’s OS, exploiting it would be virtually impossible – and even success would give the attacker only a few minutes to work before the OS would crashed. I agree with Larry Seltzer, that the point of the presentation was not the existence of a vulnerability, but the possibility of creating a “shell”. An interesting theoretical problem, not an exploit. Some of the language in the presentation makes it look like more than what it actually is.

Timmy303 • August 3, 2005 2:39 PM

The thrust of his presentation was that there was ample Cisco-fueled disinformation surrounding the exploitability of stack and heap structures on Cisco routers, this disinformation was demonstrably wrong. However intelligently designed IOS is (and portions of it really is), lax attitudes toward security code and relying on response rather than prevention can’t not lead to disaster.

What is difficult to tell from the leaked .pdf or from the presentation photographs is that he gave a POC demonstration that worked beautifully and shoveled back an enable shell at the attacker. Abaddon said there were around ten more exploitable conditions just in the IOS code he was anaylzing. The words “coming home to roost” come to mind.

Glauber Ribeiro • August 3, 2005 2:48 PM

Also something that comes up in the Wired interview, even if an attack to change the routing tables, etc is not feasible in a few minutes, an attack to destroy / bring down the router is, and the nature of the hardware makes it difficult to fix. Another very important point that is in the PDF and also in the interview, is that some of the security is gained by the weirdness of the IOS code, and that may go away when Cisco has a new version that’s better coded / not so weird.

Davi Ottenheimer • August 3, 2005 3:01 PM

Good update. I noticed the Business Week author mentions towards the end of his article “I don’t know all the details behind the story, so I may be all wet.”

He should have just started the article with the disclaimer so I wouldn’t have wasted time wondering why his perspective is so far off base.

Seltzer’s writing is not as refreshingly effervescent (pun intended) as I would have expected after his editorial comments on Domestic Terrorism:

http://www.schneier.com/blog/archives/2005/07/domestic_terror.html

His attempt to compare the situation to the Plame case is interesting, but it is hard to see how this clarifies things. For example, outing a CIA agent for political gain is a violation of federal law because it is generally the case that it seriously damages national security. Is he trying to suggest that outing a vulnerability in the Cisco IOS damages security, or even that the Plame case does not?

I do like the fact, however, that he emphasizes that we are talking about a vulnerability issue here and not a substantiated threat.

Again, Lynn’s slides indicate there is no need to panic yet, even though he publically postulated about a “Digital Pearl Harbor” and used images of the Titanic, Atom Bomb detonations, etc.. Maybe this was all just self-defence since he was facing threat of a lawsuit, but from a security perspective the imagery and fear-mongering all appear to be hugely unnecessary and a distraction from a realistic threat discussion. In my experience, the best vulnerability experts do not need to engage in this kind of hyperbole to make their point known.

While it may be true that “we’re in much more danger”, I still have not seen the true likelihood of attack or criticality discussed in a manner that easily translates to real terms, even by Cisco, to justify immediate countermeasures. Anti-virus vendors have developed a method of monitoring that distinguishes threats based on detection “in the wild”. Surely Cisco could adopt a similar system to provide perspective on the situation. That would also be far more useful to consumers than lawsuit/gag orders as it would help inform risk management decisions instead of stifling thought.

They could use Guidant’s pacemaker recall as an example, as I wrote here:

http://www.schneier.com/blog/archives/2005/07/cisco_harasses.html

Davi Ottenheimer • August 3, 2005 3:55 PM

One thing that really comes across in the Wired interview is how Lynn believes he was fighting with ISS management all along about the early disclosure. This is not uncommon from others who have left ISS that I have talked with.

He said ISS was in a difficult negotiation point and intended to “bruise” Cisco and get a leap on a vulnerability as some sort of sales/marketing ploy. Ironically, this is only vaguely different than how he portrays himself as an innocent researcher trying to bring awareness to a problem he was told to find, as if he was almost inadvertently forced into the spotlight by the circumstances.

So while Cisco knew of the vulnerability, disclosure at Black Hat seems to have happened out of confusion. One month is not exactly a lot of advance warning on a paper to be presented, especially if the build-up and relationship is as murky as described below:

“WN: So at what point did they get nervous about the talk?

Lynn: When they saw the listing of the presentation on the Black Hat site is when they actually called us back and said, ‘Wait, you guys were serious?’ And we said, ‘Yes, we were serious.’ Incidentally, it was ISS who submitted (the talk) for Black Hat. I was told (by ISS), ‘Hey, you want to go to Black Hat? We’d like you to do it.'”

But after you get through the part about ISS management trying to rattle Cisco customers comes the really interesting bit. Lynn says when ISS asked him to give the working exploit to the sales staff and pen-testers, he balked and then walked:

“At that point, I told them all no, and they fought it and I resigned right there on the spot. And this was about a month ago.

I thought they were handling this in a non-ethical manner. Because it was just way too fast and loose with who can see this…. I mean, I don’t even want people to see it now.”

So ISS asked him to reverse-engineer the code and find the flaw, and then he refused to give it to them because after he actually found it he disagreed with their sales/marketing strategy? Very interesting.

The conclusion sums it up nicely when Lynn admits he wanted “people to be afraid”. Be very afraid even though “it’s not as bad as you probably think it is. Not yet … because the version that makes this an unstoppable critical problem is not out yet.”

rob • August 3, 2005 3:58 PM

Salve Bruce!
Pictures of Michaels Slides are still here:
http://42.pl/lynn/

And what are you think about my idea of two groups (A and B) of OS/CPU/inverfaces/companies for active network elements and a new Internet architecture that every important routing can go over exclusive over one of A or B equipment so that major problems – like it is thinkable with IOS now – would not interupt all communication. IMHO it’s time for more redudancy – comments?

Davi Ottenheimer • August 3, 2005 4:16 PM

You might also want to add a link to Jennifer’s report on her personal experiences on the case:

http://www.granick.com/blog/

Some key passages:

“Lynn knew that Cisco had fixed the problem he found and stopped distributing the vulnerable code, but he was deeply concerned that the company did not do nearly enough to persuade its customers to upgrade promptly, or to explain to them why upgrading was necessary.”

[…]

“I don’t know why Lynn, ISS and Cisco were communicating so poorly. Of course, I also don’t know what Cisco and ISS were worried about, since Lynn’s presentation neither revealed confidential information nor provided much assistance to would-be intruders. Cisco also told me that they offered to give the new joint ISS and Cisco talk, but that Black Hat refused.”

[…]

“Mike wanted to show people that (1) he knew what he was talking about and (2) he could do what he said could be done. He included just enough information to make those points.”

Filias Cupio • August 4, 2005 12:07 AM

This is way off-topic, but check out the latest issue of the satirical “newspaper” The Onion (http://www.onion.com). Their “infographic” http://www.onion.com/infograph/index.php?issue=4131
is on “What are American cities doing to protect their citizens from terrorist attack?”

My two favourites are: “Los Angeles – Stationing armed, gas-masked soldiers on school buses to ensure that our children live without fear” and “Kokomo, IN – Installing lightning rods throughout the town, since chance of getting hit by lightning much greater than being killed by a terrorist”.

another_bruce • August 4, 2005 1:11 AM

haven’t these corporate dolts figured out yet that you can’t stifle interesting stuff on the internet once it’s “here, here, here, here, here, here…..”? it just makes you look scared and stupid.

Davi Ottenheimer • August 4, 2005 1:14 AM

“Is it just me, or does it seem like ISS is pursuing this out of malice?”

I think that’s a reasonable assumption from what we know about Lynn’s version of events. I have not seen the ISS side of the story yet, but one has to wonder why ISS management did not figure that they were vulnerabile to this kind of incident and do something earlier to better handle the negotiations or at least protect themselves…

Smog Farm • August 4, 2005 6:42 AM

ISS is not pursuing out of malice. They are pursuing Lynn in an attempt to keep Cisco and their large legal sledge hammers at bay.

nn • August 4, 2005 8:33 AM

“They are pursuing Lynn in an attempt to keep Cisco and their large legal sledge hammers at bay.”

Why would they do that?

H. Carvey • August 4, 2005 9:38 AM

The fact is that, regardless of all the wild, misinformed speculation going on regarding ISS’s pursuit of Mr. Lynn, the stated reason is that he’s violating trade secrets.

What Mr. Lynn did at BlackHat was divulge intellectual property owned by ISS. He may have discovered this novel method for exploiting an already-fixed vulnerability, but he did so on ISS’s time and ISS’s dime. Basically, he resigned from ISS (at least the second such time, according to some sources) and violated his employment contract. Plain and simple.

What would happen to you if you’d signed an employment contract or an NDA, and the decided to go reveal proprietary intellectual property at a conference?

This whole fiasco does nothing to further Mr. Lynn’s goals…according to his lawyer, Mr. Lynn’s original concern was that even though Cisco fixed the vulnerability, they hadn’t done enough to notify their customers. So Mr. Lynn took it upon himself to present the information at Blackhat.

How many IT admins out there know what “BlackHat” is…or even care? Sure, security-aware folks know about this situation, but it’s a mess to try to sift through. There is so much misinformation, misinterpretation (one blogger incorrectly stated that the Cisco injunction against Lynn stated that he could never present at any conference, ever again…when it specifically states that the didn’t want him to present at Blackhat or DefCon 2005), and just plain crap out there that the SNR is below 0.01%. So rather than actually notifying folks who use Cisco equipment about the issue, they’re going to be a lot of confused admins and IT managers out there.

H. Carvey
“Windows Forensics and Incident Recovery”
http://www.windows-ir.com
http://windowsir.blogspot.com

Peter • August 4, 2005 10:43 AM

Pure Malice. It isn’t the first time they’ve done so. Nor will it be the last. Cisco makes a hefty sum compelling people who’ve purchased second hand routers have to repurchase the software inside the router from Cisco. It would be reasonable to presume that more than half of the affected routers remain unpatched.

Richard Steven Hack • August 4, 2005 11:43 AM

Jennifer Grannick’s blog says the following:

“Then the Black Hat people and I double checked that the impounded official video of Lynn’s presentation was safe and sound.”

My understanding from somewhere else, however, is that a video of his talk was made or somehow acquired and that this video is already on somebody’s mini-CD and will probably be available on the Net at some point. I did a Google and couldn’t find a link, so if anybody sees it, post a comment here or to Boing Boing, please.

As for ISS trade secrets or proprietary info, Jennifer says the following:

There was the possibility that Mike had information that was secret as to ISS and that he had promised to keep secret under his employment agreement or NDA. But the complaint didn’t identify any ISS trade secrets and Mike hadn’t disclosed any ISS information other than whatever was in the presentation, so this was a great legal argument.

People keep harping on Mike’s “violation of his employment contract”.

Bovine excrement. Ruminant evacuation.

Who cares? He revealed a dangerous class of Cisco IOS vulnerabilities that Cisco AND ISS were willing to act like morons to suppress solely because they were afraid that publicly admitting the vulnerability exists would compromise their legal case if a worm appeared and they got sued by their customers. That simple (and stupid, since they’d be sued anyway.)

Cisco’s real problem, of course, is how to fix this CLASS of vulnerabilities without a major re-write of IOS AND some way to retrofit all those old routers out there that somebody purchased on eBay who Cisco doesn’t even know.

THIS is why they reacted badly – they CAN’T FIX THIS!

This is what you get when you make gobs of money by selling what are basically 486-class boxes with some custom chips and a lobotomized version of somebody else’s OS (whether TOPS or UNIX is irrelevant, but IOS looks mostly like UNIX to me) and pare down its abilities so much that when a flaw is discovered, you can’t fix it because your fix won’t fix in 16MB of memory – not that it’s rocket science, get some good assembler-language coders and fix it.

If this class of vulnerabilities doesn’t exist in earlier versions of IOS, then my point doesn’t apply. According to Mike, it DOES exist in the NEXT version of IOS. He believes it is fixable – let’s hope so.

Richard Steven Hack • August 4, 2005 11:47 AM

Also agree with Bruce’s reaction to Ira Winkler’s comments.

Winkler is totally off base. He’s basically arguing for “security through obscurity” again, which has been shot down plenty of times.

His characterization of the flaw as “already patched” and that Cisco merely wanted to prevent an exploit is nonsense. Lynn revealed a CLASS of flaws in IOS, not just the one or two specific exploits he described – which Cisco has indeed patched, supposedly.

And shutting up Mike merely allows Cisco router owners to be kept in the dark while the vulnerability info is available on hacker sites, according to Mike.

Joe A • August 4, 2005 12:12 PM

If ISS pursues legal action against Lynn, it’s not because of malice. Lynn published intellectual property belonging to ISS. ISS stands very little to gain from pursuing legal action against Lynn: they’re not likely to get much money from him, certainly not enough to justify the legal expenses and effort required to sue him. However, ISS stands to lose a lot if they do not sue him.

There is an issue of precedent: if ISS does not pursue legal action against Lynn, it is more difficult to sue others in the future. And there is an issue of making a credible threat to their other employees: if Lynn is allowed to get away with publishing proprietary information belonging to ISS, then other employees have no reason to believe that they cannot get away with publishing proprietary information belonging to ISS.

Davi Ottenheimer • August 4, 2005 1:05 PM

@ Joe A

You make several points about other reasons for ISS suing Lynn, but do not explain why malice is not one of the them.

Malice is generally related to intent to make others suffer. Mike’s interview in Wired and statements elsewhere paint a picture of managers at ISS who wanted to make Cisco suffer. What is to say they do not want to make Mike suffer as well, via a lawsuit?

“other employees have no reason to believe that they cannot get away with publishing proprietary information belonging to ISS”

That’s not true for the obvious reason that whatever Mike did can be avoided in the future by companies using better preventative controls and detective measures (see my comments above and on the earlier log entry). Jennifer also discusses this to some degree in her latest update, that Mike did not “misapropriate” any “trade secret”.

If ISS had been in negotiations for as long as Mike claims, then you would think ISS had a huge number of options to handle this in a more intelligent manner. If Mike’s description of the ISS environment is accurate, then it is easy to see why a researcher with some sense of ethics might have a conflict with managers who wanted to humiliate Cisco and exploit the exploit for profit.

Pat Cahalan • August 4, 2005 2:17 PM

Disclaimer: I haven’t read ISS’s employment agreement, and I don’t know if Lynn signed an NDA or not, or what the terms of the NDA were, or where Lynn got all of his information.

If you are employed by a company to perform research, the results of that research are intellectual property. Depending upon your employment agreement, that IP may or may not be wholly owned by your employer.

However, “business ethics” does not stop at your responsibility to follow your employment agreement (all of those who covered up the Enron accounting scandal are culpable, even though they were doing what was best for the company at the time.)

If I was doing security research, and I was told to give a presentation at a conference and scheduled that presentation, I now also have an obligation to the conference organizers to give that presentation.

If the company attempts to revoke their permission to give that presentation, I am now in an ethical dilemma -> do I follow through on my responsibilities to the business, or the responsibilities to the conference (which may include in my estimation the general public.)

If, in my judgement, the business is revoking their permission primarily in self-interest (or in transferred self-interest from another party), I probably do exactly what Lynn did.

One common argument against disclosure is, “Now someone will write an exploit, and not everyone is patched, so that’s a bad thing!”

Hackers re-engineer exploits by examining patch releases as well. From what I gather from everything I’ve read about this case, Lynn got some of his research from hacker sites. It is unclear to me how making this issue into a news item is a net loss for Cisco users -> now many more of them know that the given patch is critical and needs to be installed (as opposed to “recommended”) as soon as possible.

Look at it this way -> there are three classes of people who own/maintain Cisco gear. Those that upgrade always, those that upgrade when they deem necessary, and those that don’t do anything unless it breaks.

If Lynn doesn’t give his talk, the number of people in the second category that actually apply the patch is undoubtedly much lower than the number that do apply the patch.

Someone eventually would have written an exploit for this vulnerability. Lynn helped out all of those in the second category that have updated (or will update before such a virus exists), and had a net effect of zero on those in the other two categories (one wasn’t vulnerable anyway, and the other won’t upgrade regardless of time.) This is an overall benefit, not loss.

david • August 4, 2005 8:37 PM

I had a call from a reporter for Light Reading this evening about this. It’ll be interesting to see if they use anything I said in the piece.

mn • August 5, 2005 3:56 AM

I actually agree (in part) with Ira Winkler that the black hat attending community does not nearly represent enough cisco router users that Mr Lynn’s thought of “warning the world” is going to be effective – while at the same time the number of baddies lurking around the LV area for BH and DefCon probably far out number them.

And I just have the feeling (just guessing) that alot of people who are commenting on the problem don’t know anything about IOS, some probably haven’t even used IOS – and to those of you, comment on the actions of the players only, stop commenting on how serious/not serious the actual code vulnerability is, please.

Steven McDougall • August 7, 2005 11:38 AM

Bruce Schneier suggests that Cisco/ISS is pursuing legal action out of stupidity and/or malice. Another possiblity is that they are doing it to limit their legal liability. From a lawyer’s POV, there is Plan A and Plan B.

Plan A: do nothing.
Lynn gives his talk; no one notices. Eventually, there is an exploit (10% probability), causing large damages ($100B). The plantif’s bar files a class-action lawsuit on behalf of every company that owns a Cisco router. Cisco/ISS knew or should have known that publicizing the voulnernability would lead to an exploit, so they are liable. They can’t survive a $100B award, so they settle for $10B. Expected loss = 10% * $10B = $1B.

Plan B: muzzle Lynn, tear up the conference proceedings, smash the CDs, send C&D letters to every web site that mirros the slides. Soon, there is an exploit (90% probability) causing causing large damages ($100B). BUT…Cisco/ISS aren’t liable. Cisco already issued a patch, and Cisco/ISS did everything they could to keep the volunernability secret. They aren’t responsible for lazy sysadmins who don’t install patches. Expected loss = $0.

Schneier has argued that making vendors liable for their security failures could improve computer security. This case shows that it may not work that way. Cisco/ISS have acted to reduce their own legal liability, but they have increased the risk of compromise to the entire internet.

FX • September 1, 2005 3:11 PM

SecurityFocus published an interview (http://www.securityfocus.com/print/columnists/351) with the german hacker FX, member of phenoelit, about Cisco IOS’ security. The big thing, is that FX had a working exploit back in 2001, so all the buzz about Michael Lynn was just wrong. If you own a Cisco router, or if you would like to 0wn one, you should really read the interview. And by the way, everyone on the internet should read it as well to understand how much/less secure are routers the internet is built on…

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.