Schneier on Security
A blog covering security and security technology.
« Automatic Surveillance Via Cell Phone |
| Microsoft Permits Pirated Software to Receive Security Patches »
July 29, 2005
Cisco Harasses Security Researcher
I've written about full disclosure, and how disclosing security vulnerabilities is our best mechanism for improving security -- especially in a free-market system. (That essay is also worth reading for a general discussion of the security trade-offs.) I've also written about how security companies treat vulnerabilities as public-relations problems first and technical problems second. This week at BlackHat, security researcher Michael Lynn and Cisco demonstrated both points.
Lynn was going to present security flaws in Cisco's IOS, and Cisco went to inordinate lengths to make sure that information never got into the hands of the their consumers, the press, or the public.
Cisco threatened legal action to stop the conference's organizers from allowing a 24-year-old researcher for a rival tech firm to discuss how he says hackers could seize control of Cisco's Internet routers, which dominate the market. Cisco also instructed workers to tear 20 pages outlining the presentation from the conference program and ordered 2,000 CDs containing the presentation destroyed.
In the end, the researcher, Michael Lynn, went ahead with a presentation, describing flaws in Cisco's software that he said could allow hackers to take over corporate and government networks and the Internet, intercepting and misdirecting data communications. Mr. Lynn, wearing a white hat emblazoned with the word "Good," spoke after quitting his job at Internet Security Systems Inc. Wednesday. Mr. Lynn said he resigned because ISS executives had insisted he strike key portions of his presentation.
Not being able to censor the information, Cisco decided to act as if it were no big deal:
In a release shortly after the presentation, Cisco stated, "It is important to note that the information Lynn presented was not a disclosure of a new vulnerability or a flaw with Cisco IOS software. Lynn's research explores possible ways to expand exploitations of known security vulnerabilities impacting routers." And went on to state "Cisco believes that the information Lynn presented at the Blackhat conference today contained proprietary information and was illegally obtained." The statement also refers to the fact that Lynn stated in his presentation that he used a popular file decompressor to 'unzip' the Cisco image before reverse engineering it and finding the flaw, which is against Cisco's use agreement.
The Cisco propaganda machine is certainly working overtime this week.
The security implications of this are enormous. If companies have the power to censor information about their products they don't like, then we as consumers have less information with which to make intelligent buying decisions. If companies have the power to squelch vulnerability information about their products, then there's no incentive for them to improve security. (I've written about this in connection to physical keys and locks.) If free speech is subordinate to corporate demands, then we are all much less safe.
Full disclosure is good for society. But because it helps the bad guys as well as the good guys (see my essay on secrecy and security for more discussion of the balance), many of us have championed "responsible disclosure" guidelines that give vendors a head start in fixing vulnerabilities before they're announced.
The problem is that not all researchers follow these guidelines. And laws limiting free speech do more harm to society than good. (In any case, laws won't completely fix the problem; we can't get laws passed in every possible country security researchers live.) So the only reasonable course of action for a company is to work with researchers who alert them to vulnerabilities, but also assume that vulnerability information will sometimes be released without prior warning.
I can't imagine the discussions inside Cisco that led them to act like thugs. I can't figure out why they decided to attack Michael Lynn, BlackHat, and ISS rather than turn the situation into a public-relations success. I can't believe that they thought they could have censored the information by their actions, or even that it was a good idea.
Cisco's customers want information. They don't expect perfection, but they want to know the extent of problems and what Cisco is doing about them. They don't want to know that Cisco tries to stifle the truth:
Joseph Klein, senior security analyst at the aerospace electronic systems division for Honeywell Technology Solutions, said he helped arrange a meeting between government IT professionals and Lynn after the talk. Klein said he was furious that Cisco had been unwilling to disclose the buffer-overflow vulnerability in unpatched routers. "I can see a class-action lawsuit against Cisco coming out of this," Klein said.
ISS didn't come out of this looking very good, either:
"A few years ago it was rumored that ISS would hold back on certain things because (they're in the business of) providing solutions," [Ali-Reza] Anghaie, [a senior security engineer with an aerospace firm, who was in the audience,] said. "But now you've got full public confirmation that they'll submit to the will of a Cisco or Microsoft, and that's not fair to their customers.... If they're willing to back down and leave an employee ... out to hang, well what are they going to do for customers?"
Despite their thuggish behavior, this has been a public-relations disaster for Cisco. Now it doesn't matter what they say -- we won't believe them. We know that the public-relations department handles their security vulnerabilities, and not the engineering department. We know that they think squelching information and muzzling researchers is more important than informing the public. They could have shown that they put their customers first, but instead they demonstrated that short-sighted corporate interests are more important than being a responsible corporate citizen.
And these are the people building the hardware that runs much of our infrastructure? Somehow, I don't feel very secure right now.
EDITED TO ADD: I am impressed with Lynn's personal integrity in this matter:
When Mr. Lynn took the stage yesterday, he was introduced as speaking on a different topic, eliciting boos. But those turned to cheers when he asked, "Who wants to hear about Cisco?" As he got started, Mr. Lynn said, "What I just did means I'm about to get sued by Cisco and ISS. Not to put too fine a point on it, but bring it on."
Lynn closed his talk by directing the audience to his resume and asking if anyone could give him a job.
"In large part I had to quit to give this presentation because ISS and Cisco would rather the world be at risk, I guess," Lynn said. "They had to do what's right for their shareholders; I understand that. But I figured I needed to do what's right for the country and for the national critical infrastructure."
There's a lawsuit against him. I'll let you know if there's a legal defense fund.
EDITED TO ADD: The lawsuit has been settled. Some details:
Michael Lynn, a former ISS researcher, and the Black Hat organisers agreed to a permanent injunction barring them from further discussing the presentation Lynn gave on Wednesday. The presentation showed how attackers could take over Cisco routers, a problem that Lynn said could bring the Internet to its knees.
The injunction also requires Lynn to return any materials and disassembled code related to Cisco, according to a copy of the injunction, which was filed in US District Court for the District of Northern California. The injunction was agreed on by attorneys for Lynn, Black Hat, ISS and Cisco.
Lynn is also forbidden to make any further presentations at the Black Hat event, which ended on Thursday, or the following Defcon event. Additionally, Lynn and Black Hat have agreed never to disseminate a video made of Lynn's presentation and to deliver to Cisco any video recording made of Lynn.
My hope is that Cisco realized that continuing with this would be a public-relations disaster.
EDITED TO ADD: Lynn's BlackHat presentation is on line.
EDITED TO ADD: The FBI is getting involved.
EDITED TO ADD: The link to the presentation, above, has been replaced with a cease-and-desist letter. A copy of the presentation is now here.
Posted on July 29, 2005 at 4:35 AM
• 113 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
I don't know the legal details here, but it seems that Cisco had a stick to hit Lynn in the form of some non-disclosure agreement. It looks to me like Cisco gave an implicit permission to disassemble their code in working with Lynn in analysing the problem. Anyway, Cisco's (and ISS's) actions in this case have not been very consistent: proposing a talk to BlackHat, delivering materials, removing them from the conference proceedings...
One thing I've learned here is that a security researcher should never consent to keeping a vulnurability secret. (Whatever the policies of the company receiving the bug report are.) The moment you've accepted an NDA, the company can sit on the bug for as long as it likes.
He's brave but he'll lose a lawsuit.
His information was gained through corporate privelege; I understand Cisco's position better than his. They said: "Hey, let's make a deal. You take a look at this, we'll walk you through it, here's some tools." and ISS said: "No problem, here's one of our guys, he'll work on it."
Then the guy just goes rogue and says: "Right, I've found this smoking gun and I'm sending it all around the world! My fame and acclaim will be great!"
Cisco looks as though, not only do they write bad code, but that they are dumb enough to give it to folks who will make it public.
ISS looks as though they can't control their employees.
Michael Lynn looks like an untrustworthy gloryhound.
Instead, he should have leaked it quietly to Josh Wright or someone similar and asked them to submit it via bugtraq and Cisco's own bug tracker. Or done it anonymously himself.
This will get the problem solved more quickly and with less fuss, no lawsuits, he keeps his job, Cisco still fixes the vuln, he reports it through the ISS-Cisco partnership and ISS looks good. ISS releases a signature for their IPS in advance of the patch, Cisco announces that they've found a vuln and fixed.
What Michael Lynn did was irresponsible. He abused a position of trust.
This is that fine line between responsibilities. I understand the basic concept that "C" acted upon; "Maintain the good name". BUT the method was wholely wrong! As Mr. Schneier points out they could have turned this into a success by standing up with Lynn and saying; "Yep. It's a bad flaw but we've allocated a good amount of resources in mitigating it.". Imagine. Spending effort and money in FIXING the problem instead of the money they spend and will spend in covering it up! Really. Drop the damned lawsuit and use that money to fund the research to help close the hole!
This is what happens when the business decisions are driven by people with poor understanding of their product and its impact.
What Michael Lynn did was irresponsible. He abused a position of trust.
Didn't Cisco abuse their trust?
Whether Lynn can be charged depends largely on the ownership of the router in question, I think. A lease agreement can include any number of stipulations on usage, but once you purchase a physical object, you can do what you like with it.
Not that the Supreme Court agrees, necessarily. But this situation reminds me of something I heard from a Cory Doctorow lecture:
'That's what happened to Jon Johansen, a Norweigan teenager who wanted to watch French DVDs on his Norweigan DVD player. He and some pals wrote some code to break the CSS so that he could do so. He's a wanted man here in America; in Norway the studios put the local fuzz up to bringing him up on charges of *unlawfully trespassing upon a computer system.* When his defense asked, "Which computer has Jon trespassed upon?" the answer was: "His own."'
Did Cisco abuse their trust?
Not sure. They DID write some code with vulnerabilities in it, as has nearly every software vendor.
They then partnered with someone to analyse that code and help find those vulnerabilities, which is certainly a good thing.
The partner found some, which Cisco probably had a project to fix, but not with the priority some would have liked.
Now, that partner then lost control of one of its employees, who went public with information he gained while he was in the confidence of Cisco.
His promise of confidence has to be balanced against the risk that Cisco or ISS will act unethically with the information he has uncovered.
Do you think Cisco acted unethically?
It doesn't matter now. Lynn, Cisco, *and BlackHat*, have agreed that this won't be discussed further.
@ Nick Brooke
Wow, you're not kidding; that was quick. And Lynn looks a whole lot better in this light.
He didn't reveal any new vulns, there were indicators of a growing threat (IOS source code floating around, discussions on Chinese hacking sites).
Makes me wonder what Cisco's fuss was, then...maybe they just wanted to control the way the information was released.
It's actually completely irrelevant if Michael Lynn would have lost a lawsuit. The damage to Cisco is already done: while their reputation was degrading rapidly in the last few years, this is one more piece in the puzzle. That they were reverting to Intellectual Propoerty threats says enough. They have lost the last piece of respect I had for them.
Apart from that, what "Some guy in Europe" says is completely wrong according to the media: ISS handed over the findings in April upon which Cisco suddenly removed all older versions of IOS (that were, incidentally, vulnerable). Until the Monday before the conference ISS *AND CISCO* had planned to present *together* (again according to media, I'm currently trying to get official statements both from Cisco and ISS) when all of a sudden both Cisco and ISS pulled out.
In light of that, I don't think "Then the guy just goes rogue and says: "Right, I've found this smoking gun and I'm sending it all around the world! My fame and acclaim will be great!"" does this act do any justice.
Bruce: To me, it *smells* like lawyers deciding to cancel the talk.
What if we are talking about an automobile maker. As part of the purchase they tell you that you can't take the car appart, for the sake of protecting their IP?
Someone decides to proceed anyhow and tear appart the car and finds some problem that will cause thousands of accidents and a large number of many people's lives.
Then the auto-maker tries to sue this person for finding the problem AND making the public aware there is a problem, where-by helping the public protect themselves. Who is right?
Before you say this is computers and not cas and problems in computers don't kill people, I would caution you by saying this: MANY LIFE CRITICAL systems in hospitals may be dependant on Cisco routers.
...Just my $.02
Also...to add to my first post.
About distributing details about the IP. Don't you suppose l33t bl4[k h4ts have already decompiled Cisco's IOS and are discussing it/distributing documentation about it with others on IRC, IM, and other real time forums?
The bad guys don't care about some licensing agreement. So the good guys are to NOT make people aware of problems because of some licensing agreement which the bad people happily ignore?
"About distributing details about the IP. Don't you suppose l33t bl4[k h4ts have already decompiled Cisco's IOS and are discussing it/distributing documentation about it with others on IRC, IM, and other real time forums?"
I was at the presentation (and prefer to remain anonymous to avoid legal problems): Lynn stated that some of the information he used to develop his presentation was obtained from English translations of Chinese web sites specifically discussing IOS vulnerabilities.
some guy from Europe opined:
> His information was gained through
> corporate privelege; I understand
> Cisco's position better than his. They
> said: "Hey, let's make a deal. You take
> a look at this, we'll walk you through
> it, here's some tools." and ISS said:
> "No problem, here's one of our guys,
> he'll work on it."
That's rather serious reasoning without facts.
Keep in mind that Michael Lynn's research led them to contact Cisco with the details.
That's no different than any other security researcher finding a vulnerability (or a way to make previously untenable exploits reliable, as in this case), and disclosing to the vendor first.
Cisco has just made it clear that they will attempt to embrace the researcher closely with the legal system in response, in order to quash notification of those who really need to know -- us.
Given Cisco's multiple past uses of the broken patent system to quash fixes of problems by others (c.f. Fernando Gont), and you have the inkling that Cisco is become a monster.
The end result is that folks who find problems that affect Cisco equipment are going to have to avoid letting Cisco into the loop. That's very bad for Cisco, but they have nobody to blame for that but themselves. They've made it fairly clear that it's too dangerous to work with them.
@ Phillip Hofmeister
The problem with the car is a safety issue and should be raised immediately.
The problem with the Cisco router is a security issue.
Some folks, Bruce included, advocate full disclosure immediately for security as well as safety issues. I'm not so sure. As a professional, I'd rather discuss the issue in a trusted forum of like minded folks, figure out a tactical countermeasure and then let public discussion determine a strategic solution.
But I'm very uncomfortable about the public disclosure of a vulnerability in my infrastructure - for which I have no protection.
It strikes me that this is a risk calculation. Does anyone know how the probability of finding an exploit on the wire changes with the public announcement of a vulnerability?
I'm not so much understanding what's the propaganda there because it reallly "was not a disclosure of a new vulnerability or a flaw with Cisco IOS software. Lynn's research explores possible ways to expand exploitations of known security vulnerabilities impacting routers."
It was a damm impressive presentation. But c'mon: was anybody so far siriously thinking that it was impossible to get a shell on a machine where you could write to arbitary memory locations? Did anybody so far felt that vulnerabilities in Ciscos where less sewere since there existed no known way to extend an heap overflow into a remoote shell?
There was an interesting email on the DailyDave list mentioning rumours of this 3-way agreement, and asserting that this whole imbroglio had its genesis in informal pressure from DHS being brought to bear on Cisco (and by implication, ISS).
If the DHS part of that email is correct (as the other part was, 100%) it makes the story rather more interesting.
Mike's roommate just let me know that the FBI is investigating Mike and is currently seizing his stuff. Also, no one has any information on his whereabouts.
It's interesting that nobody has posted the real significance of this disclosure. First of all, please realize that the specific entry point that Mike was using has been patched, several months ago, so if you're running the current of IOS, you aren't specifically vulnerable to anything new. Mike disclosed the bug to Cisco, they fixed it, it's several months later, and he published it to the public.
The real significance of this analysis, though, is that due to Cisco's architecture (as discussed in the presentation) lends to the ability to use this analysis to compromise Cisco routers for any buffer overflow or heap overflow issue discovered in the future. And what that really lends itself to is that somebody could work on payload code, waiting for the next exploit to be published, then adapt the payload for the specific exploit discovered.
The real problem is that this isn't easily fixed without completely revamping the IOS architecture, or hoping, beyond hope, that no overflow is ever discovered again in IOS. Does anybody have that much confidence in Cisco that they won't have another buffer overflow in IOS in the future?
As I mentioned, a security vuln. CAN become a public safety issue very easilly.
Utility company Comminication about the power grid
Emergency services communication
Need I go on?
"What Michael Lynn did was irresponsible. He abused a position of trust.
"Didn't Cisco abuse their trust?'
Michael Lynn might very well have initially acted irresponsibly. My main point is that Cisco/ISS simply have to assume that some people will act this way, and make the best of it. Once Cisco started trying to muzzle Lynn, Lynn did the right thing. The rights of free speech are more important to society than a Cisco vulnerability.
@ Phillip Hofmeister
Sorry! I meant the safety/security dichotomy drawn in "Beyond Fear."
In his definition, safety issues are those that may occur as a result of accidental misuse or in the course of normal use.
Security issues require someone with malicious intent.
Security is concerned with, well, security issues. Security issues can make people's lives riskier - so it's still a matter of public concern either way.
Fair enough; I was definitely speculating.
"Some folks, Bruce included, advocate full disclosure immediately for security as well as safety issues."
Actually, I do not. I advocate full disclosure as a general guideline, but I also advocate responsible disclosure mechanisms. The problem is that companies like Cisco can't guarantee that everyone will follow those responsible disclosure mechanisms, and I object to them trying to enforce them through thuggery.
Perhaps Michael Lynn could have been a little more discreet in handling the vulnerability. Did he go to Cisco and tell them they needed a fix and Cisco ignored him? Or did he go off half cocked and spew to the world first? I don't know and I'll need to read up on it so I'll take the position that a vuln was found and nothing done and accept I could be wrong about Michael and Cisco. I believe I have a need to know when it comes to using a product that could have flaws. If there are flaws I want them fixed or allow me to fix them or even hire a third party to help me fix them. Simply burying the problem doesn't fix it, but covers the asses of the big guys on top. If a vuln is found by accident instead of it being known beforehand, plausible deniability can be used to the advantage of covering a mistake.
Mike (aka Abaddon) was under NDA and bound not to reveal the contents of ISS' intellectual property unless authorized to do so. He was doubly-unauthorized after he was no longer an ISS employee. This "personal integrity" does not impress me in the least. I know the details of the exploit and his expansion thereof, and it was nothing devastating enough to warrant this kind of grandstanding. His use of the term "digital Peal Harbor" didn't help any.
That having been said, I am not surprised by Cisco's reprehensible behavior, and certainly not be ISS' fawning submission to Cisco's whims. All concerned were fine with the contents of the presentation up until a few days before the conference, when they requested that the talk be removed - costing Blackhat a lot of money at the last minute to comply, and then only under the looming shadow of lawsuit. This blatant disregard for customers, conference partners, and their researchers is going to hurt them both in the near and long terms.
Guess what router company just galvanized an army of security researchers into taking their research underground, where said company can no longer control disclosure?
Guess what security research company's simpering weakness just made it very hard for them to engaging in their favorite PR posturing by giving Blackhat talks? Why would Jeff ever let them near Blackhat again after this?
Guess what recently-unemployed security researcher just demonstrated to prospective employers that he will disregard any NDA he signs any time he sees fit?
Nobody came out of this looking wise and clever. Nobody.
It's clear that Cisco has a problem.
On the one hand, they claim this vulnerability only affects older systems and has been patched. If that is true, then what's their problem with revealing it?
Obviously the problem is: unpatched routers, of which there are probably hundreds of thousands.
Which means: legal liability for Cisco.
So instead of warning every owner of said routers (if that's possible, which it really isn't), they try to prevent people from knowing about the vulnerability.
This is obviously unworkable, since the researcher pointed out it was already known from hacker Websites.
The researcher felt that if hackers know about it, EVERYBODY including the owners of said routers should know about it - and know about it NOW before the hackers release an exploit. I can't see where this is in any way irresponsible. Plenty of "responsible disclosure" was done before the Black Hat conference.
Some posters have said they aren't comfortable with knowing they have unresolved issues being revealed publicly. As stated, that is no longer the issue - hackers DO know about it. Causing a furore about revealing it isn't going to help keep it a secret, obviously. And would these posters prefer NOT to know about a flaw that hackers DO know about?
It seems fairly clear, as another poster here suggests, that the lawyers and PR people had a coniption fit that somebody would label Cisco software as a risk. I can understand that Cisco doesn't want to be perceived as allowing flawed software to run the Internet, but trying to suppress the information by ripping out pages and destroying CDs and suing people is, as Bruce rightly says, a PR disaster.
This is worse than HP claiming DMCA protection on its flaws a while back. HP backed off on that when the community reacted. The community needs to react to Cisco about this.
By the way, purportedly Lynn's discussion IS available on the Net now, so the whole "he won't talk about it any more" concession is apparently moot.
And the notion that because ISS signed an NDA Cisco has a legal "right" to sue the researcher is ridiculous. ISS has merely demonstrated it is willing to keep vulnerabilities secret on demand from a corporation. Lynn was correct in resigning and refusing to join in a censorship exercise which could result in serious repercussions for everyone else should a surprise exploit be released. ISS sued him for the same reason Cisco did - lawyers fearing they were in an untenable position should it be revealed after an exploit that they knew about it and did not inform their clients. They tried to use an NDA to avoid a possible future negligence suit - nothing more.
The stupidity is in believing that a public lawsuit would help in suppressing the revelations - or that it would help in any future such situation. Negligence is negligence. Suing somebody for revealing your negligence is hardly going to prevent future negligence suits.
As a response to the post above mine, guess which unemployed researcher has demonstrated that an unethical NDA between two companies willing to lie to the public will not restrain him from doing the right thing?
I'd hire such a person in a heartbeat.
"ISS' fawning submission to Cisco's whims"
I thought you just said that was an NDA... Which is it? An unbreakable NDA or a "whim?" You can't have it both ways.
From Boing Boing - an update:
Update 2: Randi, a reader who claims to be an ex-coworker of Lynn's, and the boyfriend of Lynn's roommate, says, "A settlement with Cisco has been reached, but ISS is still pursuing criminal charges. The press doesn’t appear to know yet that the FBI is performing an investigation now, starting with seizing equipment from Michael and his roommates. On a happy note, Mike has received quite a few job offers, including from some places you wouldn't expect."
So much for being an unemployed researcher - assuming he isn't in Guantanamo next week...
So did the guy have access to confidential information under NDA and abuse a position of trust? There's the freedom of speech but also remember you still can't do things like shout "fire" in a crowded movie theater or disclose information under NDA in the name of it. Full disclosure and its security implications are tough issues in any case.
On the subject of patching routers
Up until recently, I've worked in a network engineering capacity for a network service provider that had a network spanning most continents (I'd been at the company in question for nearly 8 years when I decided to change industries for reasons not relevant to this conversation).
Upgrading a router is not a thing which can be undertaken lightly, and in many cases, there's a definate "if it's not broken in a way that affects us, don't fix it" mentality. Among other things, code upgrades are often disruptive to network traffic (particularly on edge nodes, though it's not as bad in the core). You have to deal not only with changes made to address a security issue, but every other change made to the code between the revision your running and the new version, etc. New code inevitably introduces new bugs. New bugs occupy engineering time, can cause network problems (resulting in chargebacks, in some cases), etc. Lab testing only does so much - there's not a lab in existance that can replicate the conditions you see on a large, live network, and so there are always unexpected problems.
Cisco also has another major problem, which is their installed base. There are large numbers of cisco boxes out there on which new code images simply cannot be loaded. Period. Many of them have insufficient flash memory installed to handle the size of newer IOS images, and a large subset of these devices aren't physically capable of having sufficient flash memory installed. Past that, RAM is also a problem (you can't run an IOS Image you can't fit into memory) In short, in many cases it's simply not possible to patch such a device, short of replacing it outright, which for a variety of reasons may be neither possible nor practical (there are organizations which have hundreds, if not thousands of old, low-end cisco routers - changing equipment in these networks can literally take years)
So, to make a long story short there is a large segment of cisco devices which haven't been patched because they can't be patched. There is also a large segment of cisco devices which haven't been patched because it's highly impractical to do it. Most people would be amazed at how long a cisco box can run - I started working on them in 96, and it wouldn't surprise me in the slightest if 80% of the routers I configured that year are still running and pushing live traffic today.
So how does Cisco fix this? Pure hardware watchdogs doing system checks? Honeypots to catch the newest exploits? Revamp the entire OS? Add more integrity checks and keep upping the level of checking as new vulnerabilities are found?
"Some guy in Europe" is not as anonymous as he'd like to be. More later.
> I can't imagine the discussions inside
> Cisco that led them to act like thugs.
You must be thinking of the pre-1995 Cisco. This is entirely consistent with the behaviour of the dotcom and post-dotcom era Cisco Corporation.
Well, as for inability to patch old Cisco routers, who told them to design routers with 16MB of memory for the last ten years?
Who told them to sell what are essentially 486 boxes with some custom chips and a lobotomized version of UNIX for $3,000 or more?
Cisco has been selling overpriced boxes with minimal specs for years. It's their fault (and the companies that bought them) if the things can't be upgraded and that is now a problem for them.
I'll go further than that: it's time for Cisco to open source IOS completely. They can't be trusted to eyeball their stuff as well as outside people, and their stuff is running the Internet.
As a matter of national security, Cisco needs to open source IOS.
Otherwise somebody is going to make money putting out a 486 box with custom chips running Linux as the router - and if necessary re-implementing EIGRP as open source. I'm sure the Chinese wouldn't mind.
there is a typo in the new link to the presentation.
Time for a quiz:
Can anyone name me a time that a technology hardware manufactor re-called a product that was a product OTHER than a battery?
We left software get away with a lot of things we don't let physical devices get away with. This practice has got to stop.
> some custom chips and a lobotomized
> version of UNIX
I believe it is actually a lobotomized version of TOPS-10.
"The problem is that companies like Cisco can't guarantee that everyone will follow those responsible disclosure mechanisms, and I object to them trying to enforce them through thuggery."
I agree 100% with this statement. There is no absolute secrecy even in societies sworn to absolute secrecy. Most companies are extremely prone to leaks because the cost of establishing secrecy is prohibitive to their industry.
However, I think we also need to be careful about an eager-beaver phenomenon. There are many many reasons why someone hired by a security company and given clearance (perhaps their first job) might be motivated to violate the clearance arrangement after only a year or two. That is usually why there is a rigorous evaluation period prior to awarding any sort of significant clearance. We have to wonder if ISS is doing this due-diligence before they provide exactly the kind of experience and information that a motivated young security engineer might use to question the principles of their employer. And if this leads to a volatile relationship between ISS and their partners they really need to do self-analysis and consider why their own practices encourage their engineers to escape and disclose in the most public forum possible.
I am not saying that disclosure should be stifled through more stringent business practices, but rather that experience in security often brings perspective on when/how to disclose vulnerabilities in order to actually lessen the threat(s). I am not convinced that sound reasoning, a firm grasp on ethics, or years of difficult incident handling are prerequisites for clearance at some of these vulnerability research firms. They usually appear to just expect their researchers to be really talented at finding flaws and reporting them...
Anyway, what always stands out in most of these "hey, we're not ready yet" cases is that the person disclosing immediately has their motives questioned, and the company being exposed cries foul play. Lynn shows his predisposition to marketing hype and overstatement by calling out references to Pearl Harbor. At least he didn't go all the way to trying to create a soundbyte somehow related to Hitler. But Cisco really takes the cake for hysteria and illogical thinking because they "hired workers this week to yank related pages from handouts and substitute conference CDs". Sheesh. What better way to pour fuel on the BlackHat fire than try to overtly censor a speaker. Lynn could not have orchestrated a better press-op if he had tried.
Unfortunately all these fireworks pull us away from the real issues about trying to establish a common system and forum with clear guidelines for evaluating risk and establishing "fairness" for interested parties. It has always been somewhat implicit in the security profession what constitutes reasoned behavior, but that obviously gets tested more and more every day as more people join the profession with competing opinions and values. The courts have spent so much time debating things like "unsafe at any speed" that I expect it is only a matter of time before testimony like John's above will help real precedents be written to help us all understand when/how disclosures should be handled and who bears the cost of remediation. Back to the Cisco issue itself, it's true you have to often buy expensive hardware upgrades before you can upgrade software to patch known vulnerabilities but that is true almost everywhere in technology...and no one so far has been able to force companies to leave out new functionality in deference for security when the old harware can not accomodate the new software.
So who really gets to decide what is the "right thing to do for the country and for the national critical infrastructure". All of us? And if we are all meant to be so responsible, then are we ready to take responsibility? Should we be expected to have any related qualifications, experience, etc. and be able to confer with an independent authority that has some level of representation/validation mechanism? Incident response and vulnerability research disciplines are really still in their infancy, which is what makes security so challenging and fun.
@ R Steven;
> Well, as for inability to patch old Cisco routers, who told them to design routers with 16MB of memory for the last ten years?
Nobody I think. I cannot see any reason
why they should have added more memory
10 years ago. Memory was expensive back
then, and adding any extra MB would
decrease their profit. I think, they did
the right thing.
> Who told them to sell what are essentially 486 boxes with some custom chips and a lobotomized version of UNIX for $3,000 or more?
Nobody, but as long as nobody else
can come with a better system with lower
price (and similar ads), they will continue. And they should, too. And Why should there be faster than 486s, when the traffic does not go through the main-CPU
(I assume). I think we are paying just for
those custom chips.
No for-profit company is going to put fancy and extra HW in their product for free. Those buying routers are not as stupid as most PC buyers are (e.g. buying 3 GHz CPUs when there is no justification).
When this first broke, I was concerned that Lynn might not realize the importance of getting a GOOD lawyer.
I see I needn't be concerned about that any longer. This could get even more interesting.
@ The Professor
Cisco and Juniper both recall hardware on a fairly regular basis. They don't necessarily go out of their way to notify people, though - if you have Cisco equipment and a support contract, you can get setup on CCO to have them send you email when various configurable events happen.
In addition to being a PR problem for Cisco, this could ALSO turn in to an investor relations problem. If the investors see this move as "Cisco would prefer to cover up problems than address them"...I could see Cisco's stock price taking a big hit.
What's to tell the investors they won't do the same thing with financial problems? (See: Enron, MCI).
"Well, as for inability to patch old Cisco routers, who told them to design routers with 16MB of memory for the last ten years?"
Hindsight is 20/20, as they say, and it's never a good situation when you're using it. To your comment, they haven't been designing routers with 16 mb of memory for the last 10 years - 10 years ago, they designed a series of routers that had 16 meg of memory, and, for better or for worse, 10 years later, many of those routers are still in use. In that time, Cisco has developed routers which can be configured with substantially higher quantities of RAM, support more flash, and use faster processors. Many of their systems no longer ship with less than 64 meg of memory (which is overpriced, if you buy from them, but you can also buy from places like kingston at reasonable rates without voiding your service contract.
"Who told them to sell what are essentially 486 boxes with some custom chips and a lobotomized version of UNIX for $3,000 or more?"
Historically Cisco has used IBM chips, not Intel - as an example example, the 2500's used the old 68030's. Cisco, being the for-profit company that they are, raised the price on their equipment to the limit of what their customers were willing to pay. It also didn't help that there just weren't that many companies offering comparable hardware at that point in time.
Cisco introduced the 2501 in 1993, and end-of-saled it on April 30, 2002. Granted, the 2500's were obsolete before 2002, but Cisco continued selling them because their customers continued to purchase them. Please explain to me why a system designed in 1993, and which has been reliable enough that there are still thousands of them operational should be knocked for having a design limitation that didn't become an issue until 6 or 7 years after it was designed?
With all this talk about "he violated an NDA" and "responsible disclosure" I think it's important to note that what he disclosed was the extent to which an arbitrary vulnerability could be abused. His message was really "hurry up and put in those patches because once they're in, they can completely own you."
I think that warning people about how important it is to apply existing patches counts as "responsible disclosure." I also think it justifies violating an NDA. Lynn should be treated as a whistle-blower.
By "once they're in" I meant "once an attacker is in (because a patch *isn't* in)."
The link to the Lynn presentation is fixed. Sorry about that.
Pure speculation all around, so here is some more. The legal action in California had ISS's name on it as well as Cisco's. So why does everyone assume that ISS was pressured by Cisco? This very well might have happened, but I'm pretty sure that ISS is just as upset about what happened as Cisco. If ISS said it wasn't ready, then who knows, maybe it wasn't. Perhaps Mike wasn't happy with that decision and he wanted to go ahead and present his side of the story anyway.
Whistleblower. That's a good one. ISS and Cisco were working together on the issue and Mike didn't like the fact that he couldn't present his findings at the biggest security conference in the world. No one has even said Cisco did anything wrong. Their only crime is protecting protecting their users in a way that most techno-zealots can't understand while trying to squash a glorymonger like Lynn.
@ Chuck Gibora
"Their only crime is protecting protecting their users...."
This is the central debate. Are Cisco users better protected by ignorance or by information? In the main, I believe information is better than ignorance.
Why didn't you mention that Cisco was working with Lynn on "responsible disclosure" before Lynn decided to be irresponsible instead? You can speculate that Cisco would have prefered that this information *never* come out, but that a big assumption, considering that Cisco was initially going to participate in the presentation. Cisco didn't "muzzle" Lynn, they just asked him to wait. But he threw a tantrum, and knowing that he'd get plenty of job offers out of this, he went ahead with it anyway. There's a huge continum between responsible disclosure and standing up and handing out CD's with instructions at a conference called "Black Hat".
I'm confused why an advocate of responsible disclosure such as yourself would want to deify this guy. He's a researcher who turned on his employer and a vendor (who was actively working with him) in contravention of prior agreements simply because he wanted to release the information at this conference, and the conference was *now*. This is the opposite of responsible disclosure, and Cisco and ISS are right to explore their legal options. You are of course correct that Cisco can't control the behavior of every black hat out there, but Cisco should be able to control those under contract with them.
Yow. Not only is Cisco not looking terribly trustworthy here, but it seems like Defcon / Black Hat is losing a bit of cred as well. You go to those conferences to learn things that aren't quite public information ... unless the vendor doesn't want the information released yet, of course.
"This is the central debate. Are Cisco users better protected by ignorance or by information?"
That's a false dichotomy. The central debate is whether Lynn was correct to independently accelerate the disclosure process and release information solo at Black Hat.
Again, rather than spend too much time focusing on his or Cisco's motives I think we need to establish how to use a better system of balance so researchers are not rewarded for turning serious work into a carnival act and companies do not feel that they need to better protect themselves by hiding information from researchers, or worse.
"Responsible disclosure" means different things to different people. To most people it means that the vendor is informed quietly, given ample time to fix the problem, and then regardless of whether the vendor actually fixes their product or not, their users are warned of the flaws in the product. To those vendors, it often means "sit on the vulnerability, possibly for years, until we feel like fixing it".
I guess the question is if Cisco had been given ample warning or not. THREE WEEKS to fix a buffer overflow? A race condition might take that long to fix maybe, but Cisco was apparently just sitting on it. Cisco's users needed to be protected from Cisco's indifference to the security problems in its products.
As was mentioned above, responsible disclosure isn't even applicable here. If someone released a new egg for SPARC or Intel or PowerPC that spawned a shell or something similar, how is that different from what he did? Is the Metasploit framework illegal and does it infringe on intellectual property? No.
He didn't release code. He didn't discover a new vulnerability. He simply figured out a way to expand on existing vulnerabilities (which patches are available for) to elevate privilege and described it in broad terms. Reverse engineering is legal. Decompilers are legal. UNZIP IS LEGAL.
As a security professional, if I saw an article that Cisco devices could be overflowed I would have paid attention and said "that's interesting, another case for patch management." Cisco and ISS have spun this out of control. Great example of being paranoid about vulnerability release.
Responsible disclosure. Who ought to be bound by it? Should information security professionals and researchers be bound by it? Yes. Should information security companies be bound by it? Yes. Should software and hardware makers be bound by it? Ahhhh... there is the rub. The vulnerability of the CISCO IOS was already publicly known months ago, and CISCO knew of, and approved weeks before the Black Hat Convention, the contents of Michael Lynn's presentation. What changed CISCO's mind? We can only speculate. The abrupt and brutally effective methods that CISCO and ISS employed to silence Lynn gives the forceful impression that CISCO does not feel it is bound by the tenets of responsible disclosure. It also shows that ISS will kowtow to the likes of CISCO and sell out on its responsibilities to protecting the security of the public at large. From now on, information security professionals and researchers will labor under the threat of litigation by the likes of CISCO when they publish their findings, even though the likes of CISCO have been duly notified of the vulnerabilities. But litigation can cut both ways... especially if the current state of affairs in terms of insecurity on the Internet can be traced to the doorsteps and boardrooms of the likes of CISCO who do not feel bound by such a thing as responsible disclosure. I would not be surprised if there are many flaws in the IOS that CISCO knows about, but has not responsibly disclosed - regardless of the adverse consequences to national and personal security. It is almost a certainty that the stolen IOS code is now being analyzed for purposes of exploits - but of course, we can only wait in the darkness that the likes of CISCO and ISS have created, until it is too late.
"Their only crime is protecting protecting their users in a way that most techno-zealots can't understand while trying to squash a glorymonger like Lynn."
I don't think their users are going to understand it either. Cisco wasn't concealing just a single vulnerability so much as the level of risk associated with each vulnerability that crops up.
The presentation pdf linked above has now been replaced by a scan of a cease-and-desist letter from ISS attorneys.
Isn't this the hight of hypocrisy:
Internet Security Systems Protection Advisory
July 29, 2005
Cisco IOS Remote Code Execution Technique
ISS X-Force has discovered a technique to leverage previously disclosed
memory corruption vulnerabilities on the Cisco Internetwork Operating
System (IOS) platform to execute arbitrary code. Remote attackers may
leverage this technique to gain complete control of vulnerable Cisco
devices. The possibility of remote code execution on Cisco IOS has been
extensively researched, and this Advisory documents ISS research that
built upon these previous findings. This advisory does not document new
vulnerabilities, or recommend additional security best-practices.
ISS X-Force considers memory corruption vulnerabilities as serious
issues on all systems and platforms.
ISS Protection Strategy:
ISS has provided preemptive protection for these vulnerabilities. We recommend
that all customers apply applicable ISS product updates.
"Isn't this the hight of hypocrisy"?
Wow. It certainly is.
A coworker of Lynns has said on another mailing list that Lynn and ISS were not doing any work for cisco when the vulnerability was found. I see no semi-credible sources confirming an NDA nor a violation. It seems this may be a rumor started by a misconception on another forum? Maybe I missed something, but we shouldnt be accusing him of violating an NDA until there is any reason to believe that one existed.
"The presentation pdf linked above has now been replaced by a scan of a cease-and-desist letter from ISS attorneys."
Too late. I've got it and I'm sure thousands or tens of thousands of other people have or will have it within hours.
More moronic behavior from lawyers.
His presentation has this amusing remark:
"TCB’s I don’t know what this stands for, and neither did the people at Cisco I spoke with."
Wow - that makes me confident that Cisco knows its own OS...:-)
As for Cisco not having to put more than 16MB of memory in their routers ten years ago, sure they didn't. They could underengineer them so they could make a higher profit, so they could buy dozens of companies and make themselves into a big boy.
Look where it gets them now. They either have to recall 2500's or find some way to patch them - which frankly couldn't be THAT big a problem even in 16MB of memory - get a good coder on the job. It's not rocket science.
And when I refer to 486 boxes, I didn't necessarily mean they ran 486 CPUs - I meant level of capability. I'm aware they ran other CPUs.
Neither Cisco nor ISS had this presentation "sprung on them" at the last minute. It's not like you can walk up to the registration desk at BlackHat and say, "d00dz, I have a radical sploit, give me an hour!" According to several sources, the talk was green-lighted by both Cisco and ISS long before BlackHat.
I can personally confirm that pages 789 to 808 of the BlackHat briefing book for this year have been removed; Lynn made reference to it in his talk, jokingly. In addition, nobody got a CD until 11 a.m. on the second day of the conference, and the ones that were distributed only said "BlackHat USA 2005" on them -- no logo or anything -- so they were probably duped at some local CD duplication house as quickly and cheaply as possible. Without any reference to his material, of course.
And ISS is practically swimming in hypocrisy -- "X-Force has discovered..." That's rich. Note that they don't mention that the discovered exploit is thwarted by a Cisco patch made available in April, just pile on some FUD. I'm in a position to exert undue influence on my IT group's buying decisions, and it will be an extremely icy day in Hades before ISS even gets a sniff of our filthy lucre because of their behavior. I wonder how many other people they alienated forever by their behavior. (And it's nothing to do with whether or not there was something "confidential" there; it's all about the fact that they didn't come clean about their motives. Now they're no longer trustworthy -- the death knell in the security industry.)
Meanwhile, they can try to put the genie back into the bottle with the presentation slides. I wish he'd put up the ones he did the night before his talk; the "I Never Liked check_heaps Anyway" slide was extremely juvenile, and everyone laughed.
"Too late. I've got it and I'm sure thousands or tens of thousands of other people have or will have it within hours."
If someone puts it up out of the reach of U.S. lawyers, please let me know so I can link to it.
Let's not forget two key issues here:
1. Lynn was scheduled to co-present with a Cisco employee.
2. Conference presentations are detailed months in avance. This was clearly known both inside and outside of Cisco.
I think both of these issues speak to a larger disconnect within Cisco itself...or perhaps it was a "senior vice-president" type freakout and call to uleash the hell hounds of Cisco Legal.
Please do not ever defend the ridiculous clause supposedly prohibiting reverse engineering.
No law can remove the physical right to examine an object. This is what experimental science is all about. The Enlightenment came about when people started looking and stopped approaching issues dogmatically.
@ Heywood Jabuzzoff
Bart, is that you?
"it's all about the fact that they didn't come clean about their motives. Now they're no longer trustworthy"
I didn't think my attempt to steer clear from the discussion of motive would work. Wild speculation about motives can be so ripe and juicy, it is hard to resist.
The problem is that I have to question whether a discussion of motive adds any significant value compared to a discussion of consquences (e.g. are users better off, is Cisco worse off). I mean if we must throw darts at motives then perhaps we can sharpen the tips a bit:
Why would anyone "do the right thing"?
In Lynn's case I'll put aside divine intervention as a possibility, even though he was a former employee of the company that was founded on the principle that they could do a better job than Satan.
So, did Lynn think that his actions would pay; that taking the "right" action is always to one's own advantage? This is the Socratic Paradox, which is based upon the theory that "virtue is knowledge". In other words, if you know what is good, then you will just always do good. From that it follows that anyone who does anything wrong does not know what good is. Socrates happened to use this theory to justify attacks on other people's moral positions, since he reasoned that if they have the wrong ideas about morality or other ethical ideas then they can not be trusted to do the right thing.
Perhaps instead Lynn believed some of what I have seen quoted in the press, that he was trying to uphold the public/common good, ensuring "fair-play" by disclosing what and when he had originally intended. Some even say Cisco and ISS backed out of a done deal with regard to pre-approval of the disclosure. Many comments discuss this above, that there are a set of agreed-upon rules for disclosure and it is to everyone's advantage to follow the rules (vague and shifting as they might be). I tend to occaisonally find myself using this "Social Contract" theory whenever I am trying to ensure a lasting corporate security policy. Simply stated the idea is that a group that obeys rules is better off than one that does not. But of course this has several immediate problems including a question of who really processes experience accurately enough to arrive at proper rules and how representative they are of common goals. For example, if we consider Lynn's beef with ISS, then we see that people often consider the rules (policies) unfair and feel they deserve more than what they are being allowed to do. Even worse is when the rules are offset enough from goals that people feel obligated to break the rules to achieve what they consider "right".
I don't know if this is making any sense, so I'll just conclude with the theory of synthetic a priori reasoning, which I think rougly translates to "without the need for experience" and the opposite of where I started. Maybe Lynn just really wanted to do what he felt was the "right" thing to do, regardless of self-benefit or any system of rules. He used his insight and intelligence to reflect on the situation and felt compelled to act out in a way that he judged to be moral. In a way, he actually intended to impose a sense of what is "right" in order to avoid acting immorally, which would instead be a form of subjugation.
I'm probably just confusing things at this point, but hopefully when we discuss motive of Lynn, ISS or Cisco we can avoid trying to over-simplify and expect someone to "come clean" about motives. Or, we could forget motives all together in this thread and just consider consequences and how to arrive at the appropriate balance or trade-offs.
Theres nothing startling about any of this - companies lawyer-up and ass-cover at the drop of a hat, the interesting parts to me are a renewed discussion about (full) disclosure, and another example of the propensity for the (US) legal system to defend things that are indefensible (like legislation restricting reverse engineering, and peoples right to disclosure). As for ISS IP - to assert that the discovery of a fault is IP is almost as ludicrous as the belief that software patents are fair and moral.
True, but as you point out those two points do not reveal much on their own.
We can speculate that maybe Cisco pulled out because they originally thought they could work with Lynn, but they ultimately found themselves dealing with someone who they did not want to be associated or work with further. Partnerships fall apart for all sorts of reasons, especially with disgruntled employees. Did you see the titanic slide? Lynn certainly added a good deal of drama to spice up the presentation. Do you think Cisco reneged because they were simply not ready to present? What consequence would an exec have wanted to avoid?
Wired News has an article on the subject:
It links to a copy of the presentation here:
Which I was able to successfully download (well, at least initiate the downloading process... the file is 5.9 mb in size, and the d/l is going pretty slowly).
If it isn't already up on Freenet, it should be...
P.S. As you might guess, I'm not inclined to attach my name to a posting that says I have a copy of this - even if all they did to Richard Forno was send a cease and desist.
It's not that complicated.
Lynn helped a lot of people make more informed decisions by doing what he was hired to do, what his employer and Cisco apparently said they wanted him to do. The two companies reversed their position at the last minute and resorted to threats and tearing pages out of books.
Cisco has a PR department. If there's an excellent reason for what they did, we shouldn't have to guess.
I've uploaded the presentation to Freenet (argh, it took three tries before it went through).
Having looked through it, I can't see why Cisco is so freaked out... sure, there's some fairly detailed info about how this one particular exploit works, but it isn't like he's handing out a toolkit for the script kiddies of the world to start taking out routers left and right... we're talking Ninja level coding skills required here.
The Lynn presentation on Cryptome was last modified on 1 July 2005, thus is the one assisted by Cisco and ISS before mindchange and then ripped from the Black Hack compendium of presentations. Lynn presented a redacted version of this document at Black Hat. Compare a redaced slide from his Black Hat show with one unredacted in the PDF now on Cryptome.
Cryptome cracked the light security to remove the author's name (not Lynn) and set the last mod as July 29, 2005. And changed the filename from that used by Black Hat to "lynn-cisco.pdf."
What happened at the last minute to mindchange remains to be revealed but it is likely to do with risk of liability for the weakness which was sure to cause customer backlash when it was made public.
Could be this teacup tempest is an orchestrated "leak" of the weakness while pretending to fight release and gain a legal defense against pissed customers, a practice long-used in the fork-tongued world of national security.
If Lynn was duped by peddlers and sharks into being a suicide leaker, it wouldn't be a first time.
Here is my comment about this security problem
Cisco put a lid on the boiling pot but too late. I got to read the code just like a host of other online people who saw the Holy Grail presentation.
The horse has escaped the stable already.
John Young, I can understand your skepticism. But I doubt Lynn did this as a trick for Cisco. When people get confronted by an army of lawyers, it is very very hard to stand one's grounds.
Bruce, you did a really good posting about this stuff. Trying to weasel out the meaning of things from mere press releases is near impossible.
By the way, one of my readers gave me your URL. I am really grateful.
I put another copy on
It's hosted in Germany, outside the easy reach from the idiots^Wlawyers at Cisco.
It would be welcome if someone rewrote the presentation into a much smaller PDF, the 2MB out of PowerPoint suck.
Waiting for the cease-and desist letter from Cisco ;-). Bernd
Just look at Lynn's slides. On slide 14 he states "use a core dump image". Cisco doesn't make core dump images available to just anyone and I'd bet only under strict NDA conditions. Look at slide 13 were he wrote "dissassembly ninjitsu" - he had assistance (probably from Cisco) to pull this off. Starting on slide 18 he's disclosing Cisco source (assembly) code - code which he probably has because of the core dump version he's working with. On slide 34 he says "upcoming versions of IOS..." - who the hell is Mike Lynn to think that he can go and disclose info like that about someone else's product? That's just wrong.
The reality here is that this is all wrong. Mike Lynn saw an opportunity to screw his employer and increase his pay check. He'll get offers; but if you hire him will you trust him? If he's your employee you'll ALWAYS need to be worried that 'his judgement' is going to be the same as those signing his check.
This is probably the best example of very public irresponsible disclosure of an Internet vulnerability to date.
Since when can't US companies get injunctions in foreign courts?
"On slide 34 he says "upcoming versions of IOS..." - who the hell is Mike Lynn to think that he can go and disclose info like that about someone else's product? That's just wrong."
And you know for a fact how that this info has not yet been discussed by Cisco anywhere publicly or that it's common knowledge? In any event, if this exploit depends on these upcoming releases (which according to the presentation, it is helpful because it makes it easier), then it should be disclosed.
"Look at slide 13 were he wrote "dissassembly ninjitsu" - he had assistance (probably from Cisco) to pull this off."
We know this - it's been said that Cisco initially cooperated with ISS and supplied them with tools to assist. IIRC Lynn says so in his presentation.
"Mike Lynn saw an opportunity to screw his employer and increase his pay check."
Yeah, right - instead he resigns.
You have no clue what you're talking about. The reality here is that Cisco cooperated with ISS and Lynn until such time as some lawyer or PR person or VP decided to raise a stink.
I finally read the pdf (it merely seems to contain some information about the IOS architechture), but I still can't see what "software flaw" all the fuss is about. The only "flaw" I can imagine is if it's written in C we all know you can't write secure code with it in practice. As for the OS itself, custom operating systems for boxes like these are about always the right choice, as it'll result in faster, cleaner and more compact systems. Anyways, I still don't understand what the big secret would be here (ok, is it that they don't want anyone to know it's written in C or something?).
About the presentation, if that guy wrote it at work it means his employer owns it, not him. It looks pretty obvious to me that the guy stole it (takes his work, quits, then releases it without their consent).
Alittle late for commenting this huh. I just got back from DefCon and Blackhat, and I was in the presentation myself.
I am very disappointed at the public's reaction to this. Think for a second if you are say, the IT admin of a medium business ... would you rather some irresponsible hacker hacks your router while your vendor (big C) sits there and not do everything in its power to stop it?
REAL people can potentially (however unlikely) get hurt. What if you are a law firm and because of an exploit of your router your bona fide lawyer couldn't get to his copy of whatever important doc in time for filing so his/her poor client loses a case to some other big corp (like Mr Lynn you all anti big company crowd)?
I think big C did the right thing for its customer, to whom it has responsibility (obviously it did not do it for its share holders cause didn't this turn out to be a PR nightmare?) They pulled the IOS archive, they fixed the bug, ask yourself what else do you want from them - other than some rogue guy hack some poor IT guy who didn't know better.
I was also disappointed at Mr Lynn's "integrity". The slides presented was a modified version. He obviously submitted to pressure yet still acted in the posture of a heroic-saving-the-world figure. Don't present it and accept the boos or present the real thing then bask in the cheers, not both.
I am also alittle disappointed at Mr Schneier (still my hero!)'s canned one-size-fits-all comments on the topic...
All personal humble opinion. Yes I think it was a great presentation too, real fun, good job Michael, nice cap too.
You mention several versions of the lynn-cisco.pdf; the unmodified version from the original cd and the cracked version from cryptome, both of which are on the p2p nets.
You also state:
"Lynn presented a redacted version of this document at Black Hat. Compare a redaced slide from his Black Hat show with one unredacted in the PDF now on Cryptome.
Is there a third leaked version? If so, do you (or anyone else) have more info?
Also, this might be of interest; I couldn't find it in any Google caches (and I specified several of their datacenters), but Yahoo still has a cached copy as of this morning:
The Big Cisco Cover Up
by Michael Lynn posted June 16, 2005
Let me tell you something, I'm sick and tired of vendors getting away with lying to their customers about the scope of a vulnerability. Without honest, truthful disclosure of the full scope of a vulnerability admins have no way of knowing what is important. One company that really gets under my skin for getting away with this is Cisco. I've been told by security contacts at Cisco that what I am going to show you is impossible. You read about so many bugs in IOS that Cisco says are "just DoS attacks", and you have to wonder, is this BS? So I looked into it, and what I found is pretty scary. My work picks up where FX left off. He was on the right track, but his work always seemed too theoretical. So I busted out the disassembler and went to work. Some parts were harder than expected; most parts were easier. The end result has been that remote execution is only a little harder than most platforms, and shellcode is just a little different.
Clicked "post" instead of "preview" - oops. Gotta lay off the cheap booze.
In case I wasn't clear, the post from Mr. Lynn is no longer on the indicated blackhat.com page, nor is it in Google's cache of the page. It is in Yahoo's cached copy of the page, though. Looks like he was pissed well before the conference about how ISS and Cisco were responding to his research.
Everything I'm reading from people who understand this type of work, and especially from those who actually saw the presentation, points towards this guy as being a whistleblower who just did us all a huge favor and not some glory hog. Especially telling is the military and intelligence people praising him. The bad guys have had Cisco source for a while and are apparently exploring this area. The implications ARE scary. And Cisco did everything in their power to prevent us from knowing about it. I'm still mad about the whole "backdoor master password" incident; Cisco no longer deserves our trust or our money. ISS never did, but that's another rant for another time.
Lynn said in his presentation that some tools are "available" (e.g., public debugging tools), which makes sense if you think about the huge amount of code that makes up IOS. Particularly given that IOS is a monolith (components cannot be loaded/unloaded), this is an arduous task with the tools Cisco provide.
ISS *were not* in any kind of agreement with Cisco when the vulnerability was found. ISS routinely conduct audits without pre-approval of a company, and most of these are done with some kind of reverse engineering.
Mike Lynn found the issue while working for ISS X-Force. If you've ever worked at any kind of research firm, you're under an NDA that defines any information you uncover while working for your employer to be your employer's property.
I'll speculate on the motive, having been in security research for years. Cisco has, for years, claimed (or, at least, encouraged others to claim) buffer overflows in its code to be non-exploitable. So, when someone (ISS) threatened to shatter this marketing gem, and word got to the higher-ups, Cisco threatened a lawsuit.
Rather than face down a likely empty threat (at that point) from Cisco, ISS gave Lynn the order to withdraw the presentation.
At that point, Black Hat (acting on the official statement from ISS that research "wasn't complete") could not legally present what was still ISS property. As for why it was Cisco hiring temps, I have no idea. Perhaps because ISS was content to leave slides and information and forego the presentation.
There's no question that what Michael Lynn did thereafter broke an IPR agreement between himself and ISS.
When Lynn presented, Cisco and ISS jointly sued, both claiming distribution of proprietary information.
a) Cisco justified its suit by citing EULA provisions that ban reverse engineering.
b) ISS justified its suit by citing the IPR agreement it signed with Lynn.
Cisco could possibly have lost its suit. An agreement that prohibits me from obtaining knowledge of the engineering of my own systems is unenforceable (as evidenced by the fact that Cisco debugging tools abet such violations), unless Cisco can prove that I intended to violate its intellectual property rights (i.e., imitate or duplicate the functionality of portions of its software), because there's otherwise no real definition for what the components of reverse engineering are. Even if the RE claim had validity, Cisco should've been suing ISS, as the exercise was conducted within its labs.
Facts of the agreements Lynn allegedly broke aside, this would not be an issue if companies didn't allow such broad prohibitions to be included in the EULAs of software they bought, because one thing is certain: Lynn did the right thing.
I know I'll take heat for this, but put yourselves in Mike Lynn's shoes for a moment. I can. I've been there.
You report a vulnerability to a company. They fail to patch it. Or they sneak patches in. You know that (some portion of) their users are using vulnerable software in affected configurations. *You* feel an obligation to at least let users know that the product they're using isn't safe. There are two types of attacked users. There are stupid victims (those who knew better, or didn't take the time to know), and ignorant victims (those who weren't aware, because no information existed).
I have no sympathy for you if you're using a ten year old Cisco router in the face of the security and stability nightmare they've become. However, I'd have had sympathy for you before Lynn's announcement of the vulnerability (after all, the security impact of it was not acknowledged by Cisco). That's because you would not have had the information necessary to protect yourself.
Now you know. Patch or upgrade. If you whine about cost, you may well get owned. That's your decision to make now. You can thank Michael Lynn for that.
I personally will never buy a Cisco or ISS product or service. I will also cease auditing their code, since they have demonstrated themselves unable to handle security effectively. I can do little good. Cisco's customers in particular should know that most of the consultants out there share my view, and you will probably pay a heavy price in security because of it.
My advice to you: find a company that gives a damn about you.
What Lynn did was irresponsible.
IF his point was show this exploit is possible, he could've did it in a way that would not reveal the necessary information to actually exploit it.
There are plenty of reputable people/groups who could ethically confirm his exploit methods.
This would not only protect the _innocent_ users running older IOS, but also get his political point across.
(But maybe he wouldn't get to be a k00L k1d in the black hat club if he didn't give away the actual exploit.)
Seriously, he's just as ethical as the people who will end up creating exploit probes for the purpose of illegally breaking into a router and doing real damage.
I was at Blackhat and saw his presentation. It was one of the best presentations I have ever seen. I believe anyone who saw the presentation would agree it was given in a professional and responsible manner and frankly it is going to end up as a huge favor to us all. The given presentation was not the same presentation that is floating around on the internet now. The given presentation would not allow an attacker to quickly exploit this vulnerability. Things that Michael Lynn said that caught my ear and you will not find on the slides:
1. A significant portion of his research was done on English translations of Chinese hacking web sites.
2. This particular exploit is not a practical worm candidate because the exploit payload would have to be different for different types of routers. However Cisco is planning on moving to a new type of memory structure where the offsets would be the same for all routers, greatly increasing the chances for this or any other future discovered vulnerability to turn into a worm. A router worm, coded with a destructive payload could do far, far more damage than anything we have ever seen.
3. Cisco IOS source code has been stolen, at least twice. There is no good reason to steal this code unless you are going to attack the network. If Michael Lynn can reverse engineer this one vulnerability with IDApro, how much easier and how many more vulnerabilities can some smart Chinese attackers uncover with source code?
Now, do you think Cisco will continue to move to a new memory structure that would make router worms possible?
Knowing these 3 pieces of information, do you still think it would have served the public interest to sit on this information for 1 year?
Michael should be commended for what he did.
Do you mean he did the translations, or that he was given translations to read? Why does it matter that they are Chinese as opposed to any other "smart attacker"?
Your second point actually is in the slides.
"There is no good reason to steal this code unless you are going to attack the network."
Well of course there may be other good reasons. For example, you may want to "steal" the code to do "research" and announce your findings publically at Black Hat...
@ Matthew Murphy
You make some compelling points, but they are diminished by comments like "I'll never buy another Cisco product" and the rediculous position that victims can only be "stupid" or "ignorant". See my comments above about the Socratic Paradox...
"If you whine about cost, you may well get owned."
You may well get owned even if you do not whine. Nothing is absolutely secure. In addition, security should be managed in a much broader scope than just vulnerabilities and patches, which means layers of mitigating controls are sometimes the only solution despite the availability of a fix.
Is there an audio recording or trascript somewhere online? ;)
Were the slides at his presentations without ISS footer?
Were the slides of his presentation different from the lynn-cisco.pdf?
Where to get the 5.9 MB version of the slides?
Where are good detailed reports about his presentation?
Isn't it time to think about redudant Internet backbones/routings:
redudant against an outage of all boxes of one OS/producer/chip-family?
Will be OSS the future soon?
What to do do if a router attack will create a massive DDOS in all
@Davi other reasons to steal code:
- HuaWei has used CISCO code "unwittingly" ;)
- a competitor can discredit Cisco's "security by obscurity"
- intelligence service activities
- angry (ex-)employee
and much more - but the result will be the same: "securetiy by obscurity" is not working anymore and sooner or later the knowledge will be used.
Cryptome has two versions of the Lynn pre-Black Hat presentation, the second derived from the first by Cryptome's cracking of light security to remove the PDF author's name and change of filename from that apparently used by Black Hat to lynn-cisco.pdf. This was done to muddy the doc source (we get attempted document stings now and then, and some appear to have covertly planted trackers).
The slide from Lynn's Black Hat presentation is from a tomsnetworking.com posting by H. Cheung, a BH attendee who took photos of some of Lynn's slides. See the URL cited above:
Cryptome has on its home page a composite of a Cheung photo of a redacted slide and the same slide unredacted in the PDF.
The Cryptome PDF file size is 1.9MB not 5.9MB as a person has claimed of one version. If there is a 5.9MB version we'd like to have copy of it for comparison. Send to jya[at]pipeline.com
We are not aware of a 3rd version of Lynn's presenstation but there could be a third or more as the doc went through stages of preparation, editing, censoring, obfuscating, even being rigged for disinformation.
A full account -- video, audio, notes, recollections, whatever -- of Lynn's in-person redacted presentation would be informative. He is apparently legally prohibited from providing that so it's up to BH attendees to piecemeal the account. If these are sent to us, Cryptome will make a package of the fragments for publication unless it has been done else where. If it has been done we'd appreciate pointer(s).
I can't see the ISS logo on the photos and also some pictures (atomic explosion)
at the end is not on in the lynn-cisco.pdf. Also www.cryptome.org/lynn-cisco-2.jpg
shows a blanked part on slide 18 and the backgroud/header design of the slides
he used at BlackHat was different.
So I fear it was a disservice (Bärendienst) to Lynn to publish the slides with ISS logo (without a comment).
This _is_ relevant, because the file give the impression Michael has used ISS
copyrighted material - the first people here on this blog have judged him only
on base of this file - I fear the media will do this, too.
It could be that this file is from the CD-ROM (is it?) or a preview Michael had
send to somebody in times ISS supported him to speak at BlackHat.
When this is right - everybody who serve or link lynn-cisco.pdf should **highlight**
that this slides **wasn't** the slides he had used for his presentation!
PS: Why doesn't use cryptome (and others) hash codes to identify the documents in the wild?
"On slide 14 he states \"use a core dump image\". Cisco doesn't make core dump images available to just anyone and I'd bet only under strict NDA conditions."
You just dump your running system's memory to a file, as explained in the manual:
Do you even know IOS?
"Look at slide 13 were he wrote \"dissassembly ninjitsu\" - he had assistance (probably from Cisco) to pull this off."
Anyone can learn assembly language and systems debugging.
Do you know how to use a disassembler or even a hex editor?
"Starting on slide 18 he's disclosing Cisco source (assembly) code - code which he probably has because of the core dump version he's working with."
That's the output of the disassembler and not Cisco's source code.
Do you understand the difference?
Please send routers, switches, hubs, or any equipment to me..
13220 south 48th
There was no ISS anywhere on any slides he presented at the conference. Michael was good enough to distance himself from ISS and not let ISS take the heat. ISS did not do the same for Michael. Again the lynn-cisco.pdf is *NOT* what was presented. The screenshots at http://www.tomsnetworking.com/... are accurate of what he presented. The critical code components were blacked out. Also he had great graphics to go with each slide.
@Davi, the point about there is no good reason to steal the code other than to attack was actually Michael's words, not mine. In theory I agree that there are other reasons to steal code. However, you have got to agree the most serious danger and what you have to consider is stealing for the purpose of attacking at one's time and choosing. Finding flaws with source code is about 1000x easier than without.
I brought up the Chinese hacker part to make a point. You have to assume *This knowledge was not Mike's / Cisco's / ISS's alone*. Michael did not do the translations for Chinese people - he was researching this, found some interesting Google links and translating these posting to English it helped him do this attack. You have to assume others outside of Cisco / ISS / Michael are thinking of how to do this and even can do this technique regardless if Michael gave or never gave this presentation.
The real answer is in Rob's post. It is time to move to redudant network and Internet backbones/routings against an outage of all boxes of one OS/producer/chip-family? A worm with a 10 minute or less propogation and a destructive payload is the real threat here.
Cryptome got the Lynn/Cisco PDF after posting a call for any information about his BH presentation. It arrived overnight July28/29. At the time Cryptome posted the file around 5AM EST the 29th we had not been able to locate a version of his in-person presentation. And still have not seen much of it.
We'd like to see more information about his in-person presentation. And especially the "Chinese" hacks he refers to.
The ISS PDF version is way too corporate, too slick, too controlled, too promotional, slathered with an obnoxious logo and wee copyright hokum.
Probably a good thing Lynn dissociated himself from the over-doctored crapola which is likely to have hidden more than it revealed like Cisco's pussified security advisory.
@Bruce could you add a clear explaination about Lynn-Cisco.pdf like Neville Aga's wrote to your blog?
> EDITED TO ADD: Lynn's BlackHat presentation is on line.
It wasn't Lynn's _BH_ presetation online - it was a pre-BH presentation!
BTW the fax here http://www.infowarrior.org/users/rforno/... with the injection is interesting:
> Defendant Black Hat is hereby permanently enjoined as follows:
>1. From copying or dissemination any video recording of Lynn's July 27, 2005 presentation....
Video - has sombody just the audio of it? **g**
Someone else with a recording? Can't beleave it - hey guys you know what to do in the future ;)
It's going on:
> Hackers race to expose Cisco Internet flaw
> Andy Sullivan - Sun Jul 31, 2005 6:50 PM BST
> LAS VEGAS (Reuters) - Computer hackers worked through the weekend to expose a flaw that could allow an
> attacker to take control of the Cisco Systems routers that direct traffic across much of the Internet.
> Angered and inspired by Cisco's attempts to suppress news of the flaw earlier in the week, several computer
> security experts at the Defcon computer-security conference worked past midnight Saturday to discover and map
> out the vulnerability.
> "The reason we're doing this is because someone said you can't," said one hacker, who like the others spoke
> to Reuters on condition of anonymity. [...]
I can understud the anger - but it is not Ciscos fault that the internet is
so vulnerable - it's the reponsability of the ISPs/admins and the missing redudance!
Imagine someone died after climbing into a deep, deep cave with only one torch light - which failt and he couldn't find out.
Who would blame the bulb producer? Everybody who what to have a high available (inter)net service has take action now!
Many server use a Raid, a redudant power supply, an UPS and mulitple network connections,
now is the time to have more redudance for the internet.
So finding a new IOS exploit is not a solution, nor (IMHO) the major "to do" point,
but probably it
- would "motivate" Cisco to take more care/power to secure their products
- is needed that enough ISPs/user/admins/goverment and financial managers will start
to understand that the internet of today is (too) weak by design.
But I'm not optimistic - remember http://en.wikipedia.org/wiki/...
50 million effected people, estimated financial losses related to the outage were put at $6 billion.
AFAIK had only FirstEnergy to pay a $12-20 million monetary fine. Why should FirstEnergy and
the others invest for more supply guarantee?
Financial liability or regulations due ITU or parlaments - what else
would realy raise the (inter)net availability?
I find 1 hilarious thing out of this whole episode.
Did Cisco, a mass producer and huge developer of internetworking equipment and standards development, actually forget about the internet?!?! Did they really think they could stop the information from being distibuted and made available globally, beyond the reaches of a U.S. court? Wake up Cisco, WE ARE IN A GLOBALLY NETWORKED WORLD - YOU HELPED CREATE IT!
Cisco, defeated and brought down by the power of the internet... Such irony...
Cryptome has comments to the lynn-cisco.pdf:
# Thanks to cryptome (John Y.?) for this service and their strong spam filter ;)
Link found on Cryptome - Pictures from BH (slides):
Disclamer from Michael Lynn:
> I Am Here Representing Myself
> I am no longer an employee of Interent Security Systems (ISS)
> I do not represent ISS in any way. All opinions or viewpoints are my own and no-one**Head in the view**
Wired interview with Michael Lynn after BH:
And a look on the Trackback above is worth it - BTW is somewere a transcript of the press conference audio recording?
Abaddon absolutely did the right thing. Cisco's position that this is fixed is absolutely incorrect. What they have done is made sure that new systems are not vulnerable from the XML vector for any new equipment. They have severely underplayed the potential for disaster here and made no active effort at all to strongly encourage their federal customers fix this immediately. Shame on them for letting it get this far. I am not sure what the basis of ISS's claim that they have a fix for this is based on. Are they going to put a Proventia box in front of the router? Shame on ISS for letting a vendor sweep this under. While Cisco has a big problem with its gear and IOS, ISS has a far bigger problem in that the trust level they have developed over the years is absolutely gone. Matters of national security cannot be driven by corporate greed. It was bad enough when Enron destroyed the peoples ability to retire. Mike has made the single strongest case for open source and full disclosure. I too have known Mike for years and I am immensely proud of him. People are not harping on the real problem, that being that once virtual processes are an integral part of IOS this will be easy to script and worm.
Cisco has had problem after problem after problem with code security. As a Security Engineer I am fed-up with patching my routers every week. Cisco should fix the code. The code is broken!! Hackers and security experts find new security holes every month. What if some hacker finds a hole that the security teams don't find or that Cisco squashes the report with lawyers. We will have a very real problem, of getting hacked! Cisco, get the code fixed or we will buy competent products from Nortel, Juniper, etc, etc...
No need to get so hostile and resort to personal attacks. That just weakens your comments overall.
You do not explain WHY you have not been patching weekly. Was it because of a lack of vulnerabilities, a lack of threats, a lack of value of the assets in question, or a lack of authority to manage risk? Or none of the above.
It gets very confusing when you sidestep or completely blur the discussion relative to assets, threats and vulnerabilities.
Some say Lynn's presentation was a warning about the imminent danger (probability of attack), while others say it was evidence/proof of the flaw that makes IOS vulnerable. I have a hard time acknowledging the former because I do not see substantive/quantifiable evidence of a threat. Even Lynn seemed to say there's no need to panic yet and DefCon groupies admitted they were "busy working" on an exploit.
Compare this to something like the medical industry where threats and vulnerabilities are far more defined/regulated:
Guidant's recall is based on the fact that "the failure rate from the leakage defect will be between 0.17% and 0.51% (i.e., between 1.7 per one thousand and 5.1 per one thousand) over the remaining lifetime of the devices. It is possible that the actual failure rate will be greater than this, in part, because some past failures may not have been reported to Guidant."
Guidant has announced a voluntary recall, which obviously includes a very sensitive procedure that could introduce more risk (due to threat of infection, etc.) than living with the vulnerable system.
But perhaps the biggest distinction from the tech industry is that widespread litigation against Guidant seems practically a foregone conclusion, whereas who among the Cisco consumers ever sues Cisco for critical flaws? I'm not saying this should be the case, just pointing out a situation for reference when thinking about the Cisco dilemma.
> IMPORTANT NOTICE:
> * Cisco has determined that Cisco.com password protection has been compromised.
> * As a precautionary measure, Cisco has reset your password. To receive your new password, send a blank e-mail, from the account
> which you entered upon registration, to firstname.lastname@example.org. Account details with a new random password will be e-mailed to
> * This incident does not appear to be due to a weakness in Cisco products or technologies.
Of course, getting pictures down from tomsnetworking¹ (and others - whith help from ISS) is more important
- does Cisco keep more lawyers busy then people who cares about IT security? - SCNR
¹ see: Exploit writers team up to target Cisco routers, by R.Lemos, Security Focus, 2nd.August.2005
Cisco doesn't give out core dumps? I'm sure that you can configure the box to generate a core dump over the network using TFTP. It might require one of the hidden CLI commands to configure, but I suspect that's well-known.
The code for parsing the core dumps is part of Cygnus' distribution of GDB. I know, because I ported it to use it for core dumps from a Shiva LanRover when I worked there. A very handy thing indeed for the software developer.
Also, this lines up with bhd00d's pointer to how to use remote GDB over a Cisco console line. How the heck do you think the developers work on these boxes?
It's not like console gdb is security risk, since you should obviously protect your console ports!
Quoting from this (http://www.securityfocus.com/print/columnists/351) article: "I think it's funny because you had a working exploit in 2001, and nearly 4 years later someone (Michael Lynn) got something similar. But thanks to someone (Cisco) that chose to sue him, there was a big buzz, and all the people suddenly discovered that, "wow, IOS is exploitable, yes, you can get a shell there too". Now a lot of people want to be the first to reach the goal: make public some working shellcode." A good reading for any Cisco devices owner/0wner 8-)
I will like to know if you have or if you can get
some products like 3com router and also
the 3com-24ports. Pls kindly get back to
me with thier prices and let me know if
you can ship to The netherlands ? and for
the payment i will be paying through
credit card account.
I came upon this site while doing research on how to support Cisco Secuity appliances. That Cisco IOS has security vulnerabilities should not be a surprise to anyone with a long term technical background. Some have been in the business long enough to know where and how some of the more common vendors' software/systems originated. IOS was origniated before the Internet became what it is. Many of the issues of scaling and access environment that create some of the vulnerability issues did not exist when the software/systems were first created. All large companies have problems with not having to dump the existing successful products in favor of products that are really designed for the environment they are used in. It is a problem of not a new line of products that directly complete with the current line, with no upgrade for the current line. Another good example is Microsoft's NT OS on which the current XP OS is based. HAL, a major componant in XP was orignally writen by 3Com under the trade name of 3Open. It was a hardware direct communications services stack for headless servers operating in a "Black" network. It was not designed for the Internet and wide open computing environment that it is being used in today. That there has been some success in "closing" vulnerabilities provides "cudos" to the software engineers at Microsoft in reverse engineering their own code to make the OS something that is workable in today's computing environment. In some ways, in spite of the "hupla", the same kind of "cudos" go to Cisco software engineers.
Schneier.com is a personal website. Opinions expressed are not necessarily those of BT.