Is Penetration Testing Worth it?

There are security experts who insist penetration testing is essential for network security, and you have no hope of being secure unless you do it regularly. And there are contrarian security experts who tell you penetration testing is a waste of time; you might as well throw your money away. Both of these views are wrong. The reality of penetration testing is more complicated and nuanced.

Penetration testing is a broad term. It might mean breaking into a network to demonstrate you can. It might mean trying to break into a network to document vulnerabilities. It might involve a remote attack, physical penetration of a data center or social engineering attacks. It might use commercial or proprietary vulnerability scanning tools, or rely on skilled white-hat hackers. It might just evaluate software version numbers and patch levels, and make inferences about vulnerabilities.

It’s going to be expensive, and you’ll get a thick report when the testing is done.

And that’s the real problem. You really don’t want a thick report documenting all the ways your network is insecure. You don’t have the budget to fix them all, so the document will sit around waiting to make someone look bad. Or, even worse, it’ll be discovered in a breach lawsuit. Do you really want an opposing attorney to ask you to explain why you paid to document the security holes in your network, and then didn’t fix them? Probably the safest thing you can do with the report, after you read it, is shred it.

Given enough time and money, a pen test will find vulnerabilities; there’s no point in proving it. And if you’re not going to fix all the uncovered vulnerabilities, there’s no point uncovering them. But there is a way to do penetration testing usefully. For years I’ve been saying security consists of protection, detection and response—and you need all three to have good security. Before you can do a good job with any of these, you have to assess your security. And done right, penetration testing is a key component of a security assessment.

I like to restrict penetration testing to the most commonly exploited critical vulnerabilities, like those found on the SANS Top 20 list. If you have any of those vulnerabilities, you really need to fix them.

If you think about it, penetration testing is an odd business. Is there an analogue to it anywhere else in security? Sure, militaries run these exercises all the time, but how about in business? Do we hire burglars to try to break into our warehouses? Do we attempt to commit fraud against ourselves? No, we don’t.

Penetration testing has become big business because systems are so complicated and poorly understood. We know about burglars and kidnapping and fraud, but we don’t know about computer criminals. We don’t know what’s dangerous today, and what will be dangerous tomorrow. So we hire penetration testers in the belief they can explain it.

There are two reasons why you might want to conduct a penetration test. One, you want to know whether a certain vulnerability is present because you’re going to fix it if it is. And two, you need a big, scary report to persuade your boss to spend more money. If neither is true, I’m going to save you a lot of money by giving you this free penetration test: You’re vulnerable.

Now, go do something useful about it.

This essay appeared in the March issue of Information Security, as the first half of a point/counterpoint with Marcus Ranum. Here’s his half.

Tags: debates, essays, penetration testing, risk assessment, vulnerabilities

Posted on May 15, 2007 at 7:05 AM • 51 Comments

Comments

Clive Robinson • May 15, 2007 7:53 AM

One point you kind of missed is that in physical security as in fire prevention you know an incident it is probably going to happen. You do not know when or how or the seriousness of the outcome. So it is not possible to put 100% prevention in place.

When you accept that as a given you follow on with the attitude of,

Detect, Delay, Respond, Insure.

That is you put appropriate monertering systems in place and take reasonable precautions to stop the known atacks. Which usually takes care of 80% of the problem.

If however you have a more serious incident you rely on it being detected and that the existing systems put sufficent delay into the event so that you can respond more appropriatly (even if it is shut everything down response).

There will however always be an incident where you will not prevent a harmful outcome and that is what insurance is for. By this I do not just mean financial I include the usual stuff that people should do like Backups, Disaster Planing, etc. that will limit the effects of any damage.

Now hands up those IT officers who have actually seen the Company Disaster Plan, and have had input into it.

Ben • May 15, 2007 8:00 AM

You’re confusing a bunch of different concepts in this article, and doing everyone a disservice. The point that “contrarian” security experts make about pentesting is that you don’t need to do an all-out assault on a site to find useful vulnerabilities. Which, incidentally, seems to be your point. Pentesting is more of a no-holds-barred approach to testing where you go after everything. It’s a waste, as you effectively state, because it results in a diffuse risk analysis, whereas, again, as you point out, a more limited assessment allows more salient points that lead to a narrowed focus on key threats and vulnerabilities.

Overall, I’ve most often heard this approach called a “threat and vulnerability assessment” or similar. I’ve not heard it referred to as pentesting. The market, of course, has created much confusion by mixing and matching terms.

I think the best point of your article would have been to throw out the phrase “penetration testing” altogether, instead opting for more specific, descriptive terms and phrases wherever possible. More to the point, this underscores the importance of proper scoping in engagement agreements.

sooth_sayer • May 15, 2007 8:04 AM

Great post …. though I don’t think shredding a report is theoretically possible!! and doing that would be a bad idea.

Chris • May 15, 2007 8:27 AM

If you know your pen-testers’ skill level, and you tell them their goal is to obtain access to system X, then you can measure your performance by how long it takes them to do it.

If they can capture the flag in 30 minutes this time, but in six months it takes them 20 hours, that’s good to know.

ron • May 15, 2007 8:27 AM

IMHO, there are many analogues to penetration testing in the real world:

For example, cellular operators (or any other organization with a big call center) routinely conduct fake calls, to check how the call center people answer calls – not only whether they are polite or not, but also do they follow procedure, and do they provide information they shouldn’t give – e.g. unlisted number of somebody.

Places like Wal-Mart send testers with faked money to buy things, and test whether the cashiers actually check for counterfeit money.

Many high-security factories conduct regular checks of their guards, testing whether they follow due protocol (e.g. do they allow a courier to simply leave a package at the gate, without anybody actually coming to get that package – which is almost always strictly forbidden).

Common for all the above, and much like in IT’s penetration testing, these tests also try to find vulnerabilities in a system, by trying to circumvent some rules.

Rob R. • May 15, 2007 8:38 AM

Lots of analogies exist. In addition to those already mentioned, disaster recovery testing comes to mind.

wrc • May 15, 2007 8:40 AM

In the old days, a “penetration test” was an actual test to see if a simulated threat could compromise something of value, and how well countermeasures in place could detect or stop an attack.

That’s hard work, and requires a good team. It also requires a target that put some effort into securing their assets.

Penetration tests tell you where you are vulnerable. Everything that got compromised on the way to the goal is a “real” vulnerability. Any lines of attack that the team ignored because it took too much time or effort are where you’ve raised the bar high enough. Wherever you caught the team is a worthwhile detection capability. The places you didn’t catch them is where you should be looking.

Pen tests != vulnerability assessments != audits. Sloppy salesmanship has blurred the definition.

Ron Southworth • May 15, 2007 8:49 AM

Hi Bruce,

A baseline vulnerability assessment using various tools is effective to use in the technology security arsenal and some times as you rightly say can be over utilised. The frequency of this testing really is a matter of balancing the risk of not third party assessing the mean time to compromise your system and expending energy and resources on other aspects that will actually yield system security improvements.
I would consider investing more effort in auditing and validating system changes and patch management. Emergency / Disaster preparedness & forensic responses are some good areas to not overlook as well.

I enjoy reading your blog although sometimes the control systems info you mention from time to time is a bit out of date, perhaps but such is life is it not. Many thanks.

Mike Sherwood • May 15, 2007 9:01 AM

I’m surprised you didn’t go into the context, which seems the most important detail to me. A system dealing with mundane information that would not cause significant harm if compromised, located on a network with sensible security measures does not warrant extensive security review.

Systems with direct internet access are akin to warehouses in very bad neighborhoods. Your physical security measures are based on your threat profile. The same type of facility may have one security guard to check badges in one location and large, thick cement walls patrolled by armed guards in another. There are managers (not executives) at the company I work for that get assigned a driver to take them everywhere when they are sent to offices in some countries. The thing they’re not told unless they talk to the driver is that they’re actually an executive protection specialist – a fancy name for an armed body guard. There are some places where this level of paranoia is justified and necessary.

On the internet, there are no good neighborhoods. There are also many different networks with different security needs in any large organization. For example, an online banking web site has people trying to break into it all the time – it’s just a question of how many of them are working for the company. A site that hosts company news warrants a different level of security. It might be embarassing if someone compromised it, but it wouldn’t ruin the company.

supersnail • May 15, 2007 9:12 AM

Shame on Bruce there was no “Security is a Tradeoff ” quote here!

This is one area where the zen tradeoffs is really usefull:
1. How valuable is your data to you?
2. How valuable is your data to someone else?
3. How well have you protected your data?
4. How much are you prepared to pay make sure its protected?

If your the answers to questions 1 & 2 are “my business goes bust” and ” n thousand dollars” then its probably worth doing a penetration test to find security holes, you, can then decide if its worth the money to patch the worse ones.

Jerry • May 15, 2007 9:14 AM

“We know about burglars and kidnapping and fraud, but we don’t know about computer criminals. We don’t know what’s dangerous today, and what will be dangerous tomorrow.”

If you don’t know it doesn’t mean others don’t know either.

Brandioch Conner • May 15, 2007 9:37 AM

Bruce, you didn’t include your “attack trees” concept in this.

It’s not just whether you can be cracked. You can be. Given an attacker with enough time, money and knowledge.

But you should do an INITIAL test just to get your baseline.

Then, do another test anytime you add a potential vulnerability. Such as a website, an email server, a branch office, etc. Just so that you can show (should you ever need to) that you HAVE been proactive in searching out threats and dealing with them.

Then, at the end of every “fix” (education for avoiding social engineering, adding a firewall, fixing the website) you do another test to show that now there are fewer problems than before.

At a certain point, it will be more cost effective to just purchase insurance than to spend more money on fixing esoteric flaws. This is the point you want to get to.

That way, should you ever need to, you can show that there were financial reasons you chose the point you did and that you continued to test to make sure that you held that point.

Steve BarbedWireKiss • May 15, 2007 10:01 AM

“You really don’t want a thick report documenting all the ways your network is insecure.”

Although I could agree with most of what you have written, this one line is most fundamentally incorrect, I believe.

If you were talking from the position of the CEO or CFO of an organisation then it would almost certainly be true. But then, why the hell would you conduct an indepth (ie. technical) security analysis and then leave the report with the CEO?

I know from personal experience of setting up high visibility/traffic computing nodes that having even the briefest of a hint that there may, possibly, just ever so slightly be a “problem” that “someone” could manipulate is both an extreme worry and also a relief. Being aware that a system you’re managing is (within all reasonable conditions) “secure” means that you might be able to get a few good hours of sleep tonight. Knowing that there is “a problem” puts you in the position to remove that potential threat.

The “ostrich method of secuirty” (sticking your head in the sand so that you don’t see the problems) does not make those problems go away. There are a large number of websites and mailing lists that actually disclose potential security problems almost as soon as they’re discovered, as you know. If I’m on a list that tells me about a previously undiscovered buffer overrun in some daemon process then you can be sure that those of the black hat persuasion are also aware of that disclosure. If they know about a hole and they have a reason to try and exploit it then I think it is in MY BEST INTERESTS to repair that hole if my systems are susceptible to that hole.

After all, I’m the sysadmin, the big pointy hatted strokey bearded bloke who is supposed to be keep these systems running and make sure that they are secure and available for service. If they go down or get broken then the buck is most certainly going to end up stopping at me.

The point you raise about having the money to fix problems that are found is valid in certain circumstances, I feel. If the threat is such that your organisation would insist on some proprietary solution, or the potential target is one of the various commercial distros, then the solution is probably going to cost. With the state of the security vendor marketplace, it could cost a lot of cash. However, if you’ve been working with some of the less cash intensive systems then the real “expenditure” is time. Time, and blood, and sweat and coffee. You spend the time getting the relevant patches and bleeding them out to the server systems that need those patches. You spend the time building a secure proxy system from a spare system that was hanging around the datastore. You spend the time making sure that the last backup runs have worked and that you can fully recover levels 1,2,3,4 and 5 of the dumps to get back to where you were at the start of the day.

The reward is the satisfaction that you have done your job to the best of your capabilities and that, to within 99% certainty, you can go home, have a beer and get a good night’s sleep.

What’s the alternative? Sit around feeling that you might possibly have a security hole but it is probably not worth fixing it? Then when your systems get broken and the auditors come around and ask you what you did to mitigate the circumstances of a break, what do you say to them? “I didn’t think it was worth looking for any problems.”?

Craig Hughes • May 15, 2007 10:31 AM

“Do we hire burglars to try to break into our warehouses? Do we attempt to commit fraud against ourselves?”

Do we hire private investigators to pretext our board members’ phone records to see if anyone’s been talking to the press? Does the rent-a-cop rattle the door handle on the warehouse while he does his rounds to make sure it’s still locked? Of course there’s other testing going on all the time for other security (and non-security) systems in companies. To say there isn’t is disingenuous.

FooDooHackedYou • May 15, 2007 10:32 AM

Hey Bruce,
Interesting post. I think that if people have knowledge about security then sure they can do something about securing their systems and yes physical locations like warehouses too. 🙂 However, how many people in business really have a good understanding of security? Check out Johnny Long’s sample chapter of No-Tech hacking at http://johnny.ihackstuff.com/downloads/task,doc_details&Itemid=/gid,38/ . People actually do hire folks to penetrate physical buildings/locations as well as computer networks, etc… Have a good day d00d.

tcliu • May 15, 2007 10:33 AM

I have to disagree with Bruce on this one. Penetration testing can be very useful for protecting against social engineering.

Instead of telling your people in vague terms that “some bad people may try to trick you into doing bad things by subjecting you to stress or by other methods”, you can expose your people to such methods so that they recognize them and build up a counter-arsenal of methods.

Grant Gould • May 15, 2007 11:15 AM

I think it’s all down to attitude. If you think of penetration testing as a way of identifying all the things you need to secure, it’s clearly the wrong tool (first because that’s not possible, second because if it were possible we’d call it an audit).

If you think of penetration testing instead as something akin to pulling the fire alarm for a fire drill, then it’s much more valuable. It’s a chance to see how well you can assess, analyze, and address security flaws as they are discovered. If it takes you a week to address a newly discovered flaw rather than a month or a year, that’s a valuable sign that your procedures, staffing, and systems are on-track. If it takes a long time, it lets you discover the bottlenecks.

Security failures are inevitable; staging a real one periodically is the best way to make sure that your organization is prepared to respond quickly.

Zwack • May 15, 2007 11:29 AM

Given that this was part of Point/Counterpoint and two experts are supposed to give their Opposing views, I found it very interesting that the Opposing views are “Penetration testing is an expensive waste of time” and “Penetration testing is an expensive waste of time”.

Given that I think that Penetration testing might well be an expensive waste of time….

Dan Henage • May 15, 2007 11:45 AM

@ron: As far as analogue examples go, another one along the lines of calling in to test call centers is “secret shopping” where people are hired to test customer service by purchasing items as if they are regular customers. Also, the L.A. county fair hires students from the local colleges to go from booth to booth and make sure purchases are recorded properly with receipts, sales taxes, etc.

@tcliu: In my experience in pen-testing, companies that tell their employees that someone will call to test them on security a few times a year really helps. When you tell them not to give info to bad guys they just assume it will never happen. But when they know they are going to be tested and their boss is going to get the result, then they are more careful (whether it is a test or an actual attacker).

Unfortunately, some targets have a limited ability to resist pressure. I’ve had a few over-the-phone victims that resist for 5-10 minutes saying “This doesn’t sound right. I don’t think I should give you my password. How do I know this isn’t a security test or that you aren’t a bad guy?” To which I respond angrily: “Yes, you are right, usually you shouldn’t give out your password. That is standard company procedure. But this is not a test and we have an actual virus going around our network and if you don’t give me your password right now I can’t protect your computer and we are going to have a big problem on our hands because the virus is spreading. I have a list of 22 computers that need to be patched and you are the 18th person on the list and everyone else so far has given me their password and I’ve fixed their computers. You are giving me a really hard time and holding everything up.” Unfortunately, they cave in a lot more often than they should. But the companies that tell their people they may be tested in the future do significantly better than those that don’t.

David Thomas • May 15, 2007 12:17 PM

“For years I’ve been saying security consists of protection, detection and response–and you need all three to have good security.”

It seems to me that a big scary report could serve as a list of things you need to be ready to deal with, in one or more of these ways. You needn’t necessarily fix all of them, if you can show that you’ve been working to address them and have plans for response in case of a breach. Am I missing something?

Stacy Bresler • May 15, 2007 12:32 PM

I’m not sure I can buy into the premise that a report of risks which end up going unfixed would be so damaging to an orgnaization. If there is no business justification for accepting, mitigating or transferring the risk by the appropriate “management” that is included in the final report, sure. But we should all be working with our management to get these findings assessed from a business perspective. If the reasonable business decision is to accept the risk and that is documented in the report, then how can that be seen as damaging? Of course there are unreasonable business decisions being made all the time…perhaps, that’s the real problem.

It also seems to me that intentional ignorance (purposely not discovering your vulnerabilities) would be equally damning if not more so. Looking at the issue in the reverse, if you have become a victim of an attack and it is discovered that you did not perform any vulnerability assessment, there is a likelihood that you could be considered negligent. At least with performing a vulnerability assessment you can say that you knew about the issue; however, the business decisions was to do x,y,z based on this risk assessment and reasonable business judgment. In the latter case, you still have fault but there is documented proof that your organization thought about it and was doing what they thought was best at the time of the assessment. To me, that means less damage.

tcliu • May 15, 2007 12:33 PM

@Dan Henage: Your example of virus patching is very good.

Drilling people in what to say when confronted with this is what I mean by building up a counter-arsenal of methods. For example, telling them what to say. My experience is that a lot of the limited ability to resist pressure is that the victim simply does not know what to say! If they have a fixed script to rely on, it will be second nature.

Also, many people are afraid of their managers not backing them up if they accidentally refuse a legitimate request. As a manager, telling people that if the rules I lay down come back to bite me in the ass, then it’s my problem and not theirs, goes a long way toward fixing this.

Jay • May 15, 2007 1:24 PM

I have to chime in and say that I believe that any form of pentesting is functionally useless. While I agree with most of you that it’s nice to find where the holes in your system are, the reality is (and we all know it) that given the dynamic nature of attack methods, closing one hole will either open others or make an attacker think more ingeniously.

Network Security is fundamentally reactive, you just can’t plan for everything. That said there are measures that can be taken, but is there value in paying for or running a pentest every 6 months when it will be out of date a month from now, and can’t simulate technologies or attacks that may exist a month from now?

Bryan Feir • May 15, 2007 1:27 PM

@Dan Henage:

Yes, you’re right, that sort of social engineering would work fairly well a lot of the time.

Of course, one thing that would mitigate that would be having a common point from which any alerts like that would come. Make sure the employees know who is responsible for handling the computer maintenance, and that if anybody else comes around, be extra cautious.

Anonymous Pen Tester • May 15, 2007 2:29 PM

As a professional pen tester I have to put in my 2 cents.

As others have mentioned, there is a clear difference between a vulnerability assessment (broad coverage) and a pen test (deep).

I disagree to limit testing to the SANS (or OWASP) top 20 vulnerabilities. From experience, some of the most significant vulnerabilities in terms of consequences are outside of these lists.

When we conduct penetration testing, we usually do not generate a thick report. We document the detailed process of obtaining access – remediation is up to the client (and we may help out if necessary). For vulnerability assessments, a report is usually the outcome. But this is a question of quality of work. In a lot of instances unqualified testers do not find significant issues and – as a compensation for a lack of issues found – generate generic reports including trivial findings.

It is our experience that penetration tests are helpful for many of our clients. They may demonstrate to upper management that serious security issues exist and have real consequences.

Even though penetration testing in general is usually focused on IT security, we do not necessarily restrict testing that way. Part of our work includes physical testing of obtaining access to office buildings, call centers, etc. in order to gain access. This may include social engineering, fakes badges, etc.

IMHO If done right, both penetration tests and vulnerability assessments can be very valuable to an organization (apart from being a requirement for Sarbanes-Oxley, PCI, etc..

guvn'r • May 15, 2007 3:58 PM

Bruce, I think your comments fall into the trap of “one size fits all” security thinking.

Some years ago I worked at a site that handled approximately 20% of NASDAQ’s daily volume. Money was not an issue when it came to doing security right. More than once I was told that “cost is no issue” concerning some technical matter, in exactly those words. If the issue could put you out of business in that league, any cost was acceptable.

My assumption was that there were pen testers attempting to intrude on our systems, but I never saw them get in. The physical pen testers failed when they attempted to break in the weekend after we upgraded the physical perimeter defenses, and I’m pretty sure the same happened with the network.

In that environment it would be very useful to have friendly outsiders attacking the site, to verify that they didn’t think of something that the defenders had overlooked. To me that’s the value of pen testing, but it’s got to be more than just a glorified NMAP or NESSUS scan.

For a lot of less sophisticated sites even those might add value, if they’re run by someone more competent than the site defenders, or even just someone covering things from a different perspective. So it really depends on the site, and the testers.

David Donahue • May 15, 2007 5:54 PM

Having worked as Security architect in several large companies, I found the looming threat of penetration testing was a fantastic tool in justifying better security in the programs and networks i was helping to design/implement.

Without the boogyman of pen testing out there, If the the threat was insubstantial or hard to explain, then oftentimes the time/budget would go to increased features/reduced development time vs. better security.

While this is somewhat less true today with FUD being so prevalent; In the past, this trade off going the wrong way led to horrible designs being implemented including hard-coded SA passwords in the source code, IP address based authentication and overworked admins being assigned too many servers to keep their patches and configuration updated and their logs monitored.

I found that periodic medium-intensity Pen testing with managerial accountability for security flaws kept folks focused on making the right trade-offs.

Without some kind of enforcement, even the strongest policies get ignored in the face of budget / time crunches.

Ralph • May 15, 2007 6:02 PM

Three reasons you might do testing:

One – Education
Many people don’t have a clue about how to secure their assets and this can be a way to start an assessment and providing an intro and simple structure to the process.

Two – Compliance
You don’t have a choice, the auditor is an idiot and wants his bit of paper regardless how pointless.

Three – Comissioning
You’re a large organisation. The web site was developed by another department in an outsource project; the programmers were contracted and are now gone. They give the box to you and say take it live. Testing can form a minimum standard to reach before you’ll put it in your data centre.

Ralph • May 15, 2007 6:03 PM

Three reasons you might do testing:

One – Education
Many people don’t have a clue about how to secure their assets and this can be a way to start an assessment and providing an intro and simple structure to the process.

Two – Compliance
You don’t have a choice, the auditor is an idiot and wants his bit of paper regardless how pointless.

bill marriott • May 15, 2007 10:39 PM

I see the point of both views here, but I think you are missing the bigger concept – outside of the purely technical discussion. Perhaps we can channel Former President Reagan – trust, but verify. A pentest is essentially just an audit. Audits are performed all the time, not just in technical arenas. The objective is to determine the level that controls are acting as intended (specifically in this discussion, that systems are as secured as we intend). We have audits performed on all sorts of areas outside of the purely technical – financial processing, disaster recovery tests, emergency exit strategies, on and on. Its part of managing people, you want to trust that they are doing the right thing, but sometimes it’s valuable to make sure. It goes to accountability.
I think we, on this list, can agree that there is no such things as an impenetrable system security. It is really a matter of time vs. processing power vs. patience. If you give someone long enough, with a powerful enough system, they can get in (assuming good skills). However, there is a lot to be said when I can break into your website in 20 seconds versus 20 days. Sure, some targets will be more interesting than others (bank transfers vs my local YMCA) and people will spend longer trying to break in, but that’s part of your risk evaluation for each particular system or set of systems and the controls (both technical and process) that are applied.
Audits can be used to see whether your controls are working as you intend. If you think you have made good decisions and spent money wisely in protecting your systems, with good controls – both technical and process related, than its valuable to have someone kick the tires to check it out. Tie good technical security with good incident management process and you may keep most out at the door, but if they do get in (and given enough time they will), you can control the damage.

Peter Hentges • May 16, 2007 12:21 AM

“Do we hire burglars to try to break into our warehouses? Do we attempt to commit fraud against ourselves?”

We do in the world of movie-plot threats! See Sneakers (http://imdb.com/title/tt0105435/).

RonK • May 16, 2007 3:57 AM

@ Zwack

Even more interesting is that one of the two experts sells automatic penetration testing software. Weird. I suppose he was talking about paying a human to do it…

Joris • May 16, 2007 6:10 AM

I agree with ‘Anonymous Pen Tester at May 15, 2007 02:29 PM’ that limiting a test to the SANS Top 20 list is not a good idea. What will your conclusion be? You’re patch-policy failed? And “implement a good patch-policy’ will be your advise? Than that’s my free pentest 😉

Of course you want to know if you’re vulnerable to the most common vulnerabilities, but i don’t think companies need a pentester to scan for the ‘top 20’. But you alsso want to know if your systems are vulnerable at all. How vulnerable is your network design? How vulnerablle are youur applications?

These are not the questions your customer will ask for. He will ask you: “How vulnerable am I, and what can I do about it?” He doesn’t care if an attacker used a ‘top 20’ or a vulnerability in an custom-made application.

David Gamey • May 16, 2007 6:14 AM

Bruce I read your post on Pen Testing with interest.

As a practitioner, I find a lot of what passes for pen testing serves the testers more than the customer.

I’ve also seen requests for pen tests from customers that yell sucker or responses that were oleaginous. A couple of my favorites were:

From an RFP “Proponent will find and exploit worm holes in system security” (I was sorely tempted to write in Steven Hawking into the response). Oh, yes there was also a do no harm requirement in this one.
In an RFP response “Our team uses industry standard exploits to ….” (translation we download from the wild and let fly).

It’s not that elaborate tests aren’t needed. It’s just that most of the time they’re overkill. Get the basics right, figure out how it all fits together, then let your tolerance for risk and your situation be your guide.

Thanks
David

Ronald van den Heetkamp • May 16, 2007 7:24 AM

May I pentest your site Bruce? You use “Movable type” which has vulnerabilities.

So frankly I don’t get what you are trying to say here.

You are vulnerable too.

Ronald van den Heetkamp • May 16, 2007 7:26 AM

May I pentest your site Bruce? You use “Movable type” which has vulnerabilities.

So frankly I don’t get what you are trying to say here.

You are vulnerable too.

Bruce Schneier • May 16, 2007 1:41 PM

“May I pentest your site Bruce? You use ‘Movable type’ which has vulnerabilities. So frankly I don’t get what you are trying to say here. You are vulnerable too.”

I know I am. “Bruce Schneier Facts” to the contrary, I don’t have any magical ability to make things secure.

John • May 16, 2007 2:01 PM

Doesn’t the TSA get pen tested on a regular basis.

With scripted items?

And still fail?

Miserably?

And steadfastly NOT correct the deficiencies?

While focusing on shoes, hair gel, and toothpaste?

Right.

Ironox • May 16, 2007 2:27 PM

As mentioned once above, a pen test is just another kind of. Any decent financial audit will check for fraud loopholes in the business process, as well as for indications of active fraud. It’s just that financial audits have had a LOT longer to develop industry standards and best practices. And the cost/depth of any audit should be a function of acceptable level of risk – bank audits are in practice readily distinguishable from small business audits, though the principles are the same.

Anonymous • May 16, 2007 2:30 PM

The purpose of a penetration test or a vulnerability test is to give you an idea of how to prioritize the remediation efforts for the vulnerabilities that do exist. It is part of a process and should not be done on its own. You need to:
-validate the cost of performing any tests,
-identify the vulnerabilities,
-perform a threat risk assessment,
-conduct a cost benefit analysis,
-align remediation efforts with business requirements,
-execute remediation efforts.
This is an expensive process, but if it is done intelligently, the total costs will be less than the total costs of losing the information should an incident occur. If at any time the cost of following this process exceeds the value of the information to the business then you should stop and rethink what you are doing.

Dan Klinedinst • May 16, 2007 3:40 PM

After my initial read, I disagreed with everything Bruce said. As many have pointed out, we do test other business processes, via audits, DR exercises, QA tests, etc. Documenting vulnerabilities is not a waste of time; it’s the only way to accurately measure risk. Focus on the SANS Top 20? We hire pen testers to find vulnerabilities we don’t know about.

On second thought, though, what I’m thinking about are vulnerability assessments. I do want to know what threats my network is exposed to. Only then can I determine if the cost to fix each vulnerability is less than the cost of leaving it exposed. However, pen tesing usually connotes attempting to “hack in” via any possible attack vector. Often this results in an emphasis on “getting in” rather than on thoroughly documenting as many vulnerabilities as possible. That type of pen test is much less useful, as well as being potentially disruptive to your business. As a colorful metaphor, a vulnerability assessment is like walking around your warehouse, trying the doors, testing the alarm, and making sure the guard is awake. A pen test is more like watching a SWAT team blast their way through the front door. Which is only useful if your warehouse expects to face that sort of threat (and the contents are valuable enough to justify defending against it.)

Ranum Fan • May 16, 2007 4:06 PM

I am curious, does anyone do pen testing not just to see how bad the network is but to see if the systems that are suppose to alert/stop security issues (IDS, Firewalls, IPS, central logging, anti-virus, admins reading logs and log reports, change control, automated policies (profiles), etc.) actually work in the enterprise? If they don’t take advantage of using the time to also verify if those systems work, then ya it does sound like pen-testers are a black hole of your budget. Penetrate, read report, remain clueless, repeat. Hey this is just like the securitymetrics.org “Hamster Wheels of Pain”

http://www.securitymetrics.org/content/Wiki.jsp?page=Welcome_blogentry_040505_1
http://www.securitymetrics.org/content/Wiki.jsp?page=Welcome_blogentry_061005_1

Jesse • May 17, 2007 1:39 AM

Another reason to pentest is to aid in getting rid of an IT employee.

Roger • May 17, 2007 6:43 AM

Bruce wrote: “Sure, militaries run these exercises all the time, but how about in business? Do we hire burglars to try to break into our warehouses?”

Minor nitpick, but both of these sentences are wrong or misleading. The DoE periodically conducts force-on-force tests at some nuclear facilities, and they occasionally use military personnel for this. However TTBMK&E the military itself very rarely conducts actual pen tests; contrary to popular myth, the purpose of most military exercises is to provide training to the participants, not to assess the effectiveness of any given defence. Further, DoE force-on-force exercises are not true pen tests in the IT sense bcause the nature of the exercise requires the facility guards to be warned that something is about to happen (their live ammo is replaced with blank, and a “shadow guard” arrives to take over the actual security while the exercise is taking place.)

As for hiring burglars to break into things, yes! UL does exactly that to produce its security ratings. The difference I suppose is that UL testing is done on products rather than specific installations.

You also wrote: “And two, you need a big, scary report to persuade your boss to spend more money.”

I think that just about nails the main reason for them. They focus attention on security. Normally management regards information security as a pure loss centre driven by paranoid propeller heads. When you can report that a hired cracker penetrated your systems in 12 hours and got access to the boss’ computer, the boss sits up and pays attention. In this sense, they may be security theatre, but they are security theatre being used for good; at least, more proportionate and more targetted than anti-virus companies constantly shrieking that the sky is falling!

However, they also serve to keep everyone honest. Every other form of security audit I have seen can be — and often is — “baffled with bullshit” i.e. faked out with voluminous, correctly filled out paperwork that has only as much marginal contact with reality as necessary to avoid being easily caught out, e.g. documented answer: “critical documents are shredded before disposal”; truth: “We have a shredder, but no idea if anyone uses it”. Even when compiled with the best of intentions, in a large organisation there may exist several level of filtration and abstraction between reality and the report, and anyone along the chain who doesn’t understand a concept, or makes an assumption because he is too overworked to check, renders the whole thing meaningless. A pen test, on the other hand, ACTUALLY TESTS the security system. It may be an imperfect test, even a seriously defective one, but at least it is a real test rather than a paper audit.

Ronald van den Heetkamp • May 17, 2007 8:07 AM

@Bruce who said:

…[I know I am. “Bruce Schneier Facts” to the contrary, I don’t have any magical ability to make things secure.]…

Okay, so why say others must secure things? It’s possible to write secure blog software. It ain’t that hard to do.

tcliu • May 17, 2007 2:19 PM

@Ronald: It’s possible to write secure blog software. It ain’t that hard to do.

Hop to it, mate.

mjw • June 18, 2007 6:09 PM

I’m coming into this debate quite late, but I’m not so sure I’d so readily dismiss the usefulness of a pen test. Sure, you can have terrible testers but I think there are those of us that add a great deal of value to our customers.

When I perform a penetration test I strive to look for the low hanging fruit. I have experience with the DoD, IC, and other top tier organizations. Ripping apart the average company doesn’t boost my ego or provide much benefit to them. I want to raise their security a notch. I want to figure out who their adversary is and try to attack like he would. I go for the easiest way in. I then meet with the stake holders, discuss the enablers, show the impact of my attack, and help them begin mitigation. Yes, there’s the obligatory report that’s file away on a shelf for some auditing requirement, but it’s secondary. When I leave the customer is fixing their system and the bad guys have to a new set of hurdles.

Driften • June 27, 2007 12:43 AM

Just wanted to add my late 2 cents worth. I agree with Bruce and think if you don’t want to know the answer you should not ask the question.

I think a Pentest can be great but you really need to take a tactical approach and know you can’t ever be totally secure but use it to make sure the high risk items are tested shown to be fixed/secure. I also sure would not want a report saying “I told you so” around if a security hole reported in the test was exploited and you knew about it.

Passive listener • September 10, 2007 10:17 PM

This has been an interesting exchange of comments and opinions. In the company I work in, the security team thinks that pentest is one of the initiatives to roll out in the new FY, and I am tasked with formulating the “roadmap” to bring it into our environment.

I am trying to convince myself about the pros of starting a pentest program in my company. The problem I have is whether at the end of the day, pentest does give us value. No doubt proper planning and assessment is required before we can even decide where to pentest. However, I fear that in the end, the pentest will just tell us that we are vulnerable and that I know already.

Augeas • November 7, 2007 11:13 AM

The Penetration testing term is used in a variety of ways as mentioned in a few of the posts above.

Traditional penetration testing involves attempts to exploit a network to uncover all vulnerabilities that can be found within the network in a fixed amount of time. Typically, everyone knows you are coming and no efforts are made to go “low slow” and keep the testing activities under the radar. The discovery of the vulnerabilities is the first step, the actual impact of the vulnerability discovered on the rest of the network is where the value lies.

For instance, once a single vulnerability is discovered and exploited the fix may be a patch or security control setting, the impact on the computer/application with the vulnerability may be known, but the impact on the network is still unknown. Through discovery and exploitation, penetration testing can reveal the impact to the entire network based on the single vulnerability being discovered (continued testing using the initially exploited server as an entrance point into the network).

A proper analysis of the results will provide an organization with a cost/benefit based plan to mitigate the vulnerabilities within their network, and will also uncover process/procedural issues as well.

A properly performed baseline analysis and evaluation will provide an organization with a plan to correct the discovered vulnerabilities and proactive mitigation of future vulnerabilities.

Penetration Testing • August 15, 2008 4:43 AM

Penetration Testing does provide value but only when used properly. There’s definitely a difference between testing for testing’s sake (obviously wrong) and other means of penetration testing (e.g. to develop an action plan, to use as a means of demonstrating compliance with standards etc.) when it’s required.

The analogy you’ve put forward regarding burglers is somewhat flawed. A better analogy would be to look at the aviation technology industry. Software is subject to extreme scrutiny due to high assurance requirements. Hardware is tested for all manner of issues with both positive and negative approaches taken.

Is Penetration Testing Worth it?

Comments

Leave a comment Cancel reply