Research on Patch Deployment

New research indicates that it's very hard to completely patch systems against vulnerabilities:

It turns out that it may not be that easy to patch vulnerabilities completely. Using WINE, we analyzed the patch deployment process for 1,593 vulnerabilities from 10 Windows client applications, on 8.4 million hosts worldwide [Oakland 2015]. We found that a host may be affected by multiple instances of the same vulnerability, because the vulnerable program is installed in several directories or because the vulnerability is in a shared library distributed with several applications. For example, CVE-2011-0611 affected both the Adobe Flash Player and Adobe Reader (Reader includes a library for playing .swf objects embedded in a PDF). Because updates for the two products were distributed using different channels, the vulnerable host population decreased at different rates, as illustrated in the figure on the left. For Reader patching started 9 days after disclosure (after patch for CVE-2011-0611 was bundled with another patch in a new Reader release), and the update reached 50% of the vulnerable hosts after 152 days.

For Flash patching started earlier, 3 days after disclosure, but the patching rate soon dropped (a second patching wave, suggested by the inflection in the curve after 43 days, eventually subsided as well). Perhaps for this reason, CVE-2011-0611 was frequently targeted by exploits in 2011, using both the .swf and PDF vectors.

Paper.

Posted on May 20, 2015 at 2:15 PM • 15 Comments

Comments

Vulnerability ResearcherMay 20, 2015 4:58 PM

This is a very well known problem in application security areas, which, btw, are often separated from the same organizations in charge of patching at many organizations. :/ That separation is always a bad combination. But, usually, an organization's it security department will have some visibility and at the very least be in charge of operating security scanners. (For whatever reason, from corporate to government, these organizations seem to naturally evolve out into the very same structures.)

VeraCode actually does a pretty good job of ferreting out problematic third party libraries that have vulnerbilities. But they have a high false positive rate in general, like IBM AppScan Source & HP Fortify.

That is one of the most insidious problems with patching: third party libraries which are included in custom vendor code. Third party libraries are a staple in the industry, that will not change. Taking code from online sources is a related problem, and also that practice is likely not to change. Typically, however, that can be even more insidious when they are security vulnerabilities in that manner of code. Because that kind of code is very often put in by patch work and so more difficult to detect.

Much of these problems are in the grasp of network scanners. Some of them perform analysis simply by authenticated registry calls. Others perform full binary sweeps checking hash data of known bad vulnerabilities. The later is obviously much stronger at detection for known vulnerabilities... but it is far more intrusive. And being more intrusive means something: already full AV scans are extremely demanding on systems, and this sort of functionality operates in much the same way.

Patching, I do believe, one can reason out is, however, the best strategy to have. The problem is heuristic technology for detection and defeating of vulnerabilities already known is sketchy and highly resource intensive. This later strategy can be, however, invaluable to properly detect who is attempting 'what' attacks, and so promises to expand, greatly, the capacity for maintaining accuracy against well known attacks.

Why are those sorts of systems sketchy? Because there are many classes of critical vulnerabilities which can have an extensive array of potential exploit code and much of the exploit code might appear little or no different from legitimate activity. This is, for instance, especially true with 'business logic' type vulnerabilities, and many forms of web application vulnerabilities.

It is entirely troubling when it involves lateral network sensors, because most security strategies are designed for perimeter defense, not lateral, 'behind the DMZ defense'.

This later problem is increasingly problematic as companies increasingly embrace 'iot' technology. Already companies are deeply invested in wide spread wireless access and there are many problems with that access to where they effectively mean that their users and customers are bypassing ordinary dmz perimeter controls. eg, handset applications which directly communicate with post-dmz databases, or internal applications initially designed for wired network security instances but now effectively open to external inspection simply because internal access goes over the air, as opposed to the wire.


theodoreMay 20, 2015 8:51 PM

No one patches any more. The former "Patch Tuesday" should have really been called "Replace Tuesday." I'll always remember Chuck Knight and Ray Scott patching software while it was running. That's when I knew that I wasn't a programmer.

AnuraMay 20, 2015 8:57 PM

The biggest problem with closed source is indeed the problems with patching. With open source, everything can be built together, tested, and distributed individually by a single package manager in a central repository, ensuring that everything gets updated. If you have a closed source application, you can run into dependency issues. If you depend on a library, and that library may be updated independently of your software, then it can break it. One way around that, which a lot of companies choose, is to distribute the library with your binaries either by static linking or including the shared library. Now, if you don't update that library you included, you will be vulnerable. If there is a breaking change, many businesses will choose to stop updating the library rather than taking the time to make their code work. These days, windows supports side-by-side assemblies which allows multiple versions of the same DLL to be installed so they can be shared, but it can have the same problems when it comes to patching as all dependent software needs to be updated.

Dealing with multiple vendors is a pain in the ass.

FigureitoutMay 20, 2015 9:15 PM

The little blurp states why just "patching" is insufficient if say, what if there's a bug in the patching software itself or using an emulator WINE like these researchers did, does not give a true picture of reality and has security issues itself. Or what if, as basically all patches do, the patch is applied via internet and there's a malware on your router that fakes upgrades (does anyone remember how untrustworthy MS update was?) or malware that doesn't allow upgrades. You can be damn sure the build environment of Flash Player or PDF readers are going to be absolutely massive pieces of ... that no one can wrap their head around and "it just works, for now".

Patching is probably as it sounds, putting a band aid that peels off after a week on what may or may not be bigger problems.

Vulnerability ResearcherMay 20, 2015 10:49 PM

@Figureitout

The little blurp states why just "patching" is insufficient if say, what if there's a bug in the patching software itself or using an emulator WINE like these researchers did, does not give a true picture of reality and has security issues itself.

:-)

Unfortunately, for everyday corporations *not* super paranoid government, dealing with the potential threat of attacks via your security scanner is just not on the map. There is so much else to worry about.

I smile, however, because looking for security vulnerabilities in security products is *exactly* what some of my friends have done. They went the responsible disclosure route, but it was eye opening activity.

I also agree, on the WINE comment, and that stood out to me. You want your studies performed as 'real world' as possible.

Vulnerability ResearcherMay 20, 2015 10:52 PM

@Anura

Good point on open source libraries, and have had to hack some up before. Though, very often, developers do not have time for that.

A lot of code these days is not even delivered directly to consumers. So the choice never even comes up. eg, web applications, server applications, internal customer applications, web services, etc.

FigureitoutMay 20, 2015 11:50 PM

Vulnerability Researcher
:-(
*not* super paranoid government
--I've seen some different things (while I enjoyed some nice access, most times it's probably too boring or not worth it); at the very least if they don't change protocols then that info can be sold or given away and people can know schedules etc. and start building up a safe attack chain if they care (the more intel, the better, means no risky reconnaissance where sh*tty attackers will get caught)...And the security holes will sneak in since no one company can deliver on everything and you still use Windows right..?...But...
There is so much else to worry about.
--Yep, many times not specifically security related (unless all you do is sell "security" which is too big a subject to be one word considering attack space now).

Vulnerability ResearcherMay 21, 2015 1:48 AM

@Figureitout

I've seen some different things (while I enjoyed some nice access, most times it's probably too boring or not worth it)

I like to study crime and intelligence. But for all the vast number of secrets kept by the nations, I have yet to come across anything interesting in and of its' self.

:-) People are not interesting because of the actual knowledge they have, but because of the knowledge they **think** they have. :-)

The drama they create. Over nothing. It is constantly interesting. To this, you must admit. :-)

I can not tell you how many times I have been drenched with sweat over nothing, or angry about... well? Just a stage show.

But, damned. Someone, please. Applaud our efforts. Even if our directions are so absurdly wrong.

And the security holes will sneak in since no one company can deliver on everything and you still use Windows right..?...But...

Heh, I do, but I am a walking honeypot.

I could care less if anyone gets at any of my secrets, if they knew anything it would only make their brains bleed through their nose.

My song is from some stupid Nicholas Cage movie back in the 90s.

If people really want to test their own capacity to keep their own sanity, well go for it. Because I am doubtful of their capacity.

Being so lonely and misunderstood is always a position that one wants to share. Even if that means it will only traumatize anyone who does know.

Yep, many times not specifically security related (unless all you do is sell "security" which is too big a subject to be one word considering attack space now).

Information Technology, IT, industry.

Our brains are online now, effectively. There are no more secrets.

Complete transparency for everyone. Only those behind that and most gung ho think that means, "For everyone but me. And us."

They are quite incorrect.

Everyday people's secrets are not very interesting. They rarely betray or feed of anyone.

What is really interesting are the secrets of those who feed off and betray the everyday people.

Because they do betray them and so their own self. And because they do feed off them. And for what? They would have been better off as impoverished meth heads if they wanted a temporary high.

People whom they could not exalt themselves more over.

It is all about evening out the score. What better movie could there be then that.

It is true crime, and everyone likes that. See the mighty fall for the very reason of their being not so mighty to begin with. I mean, 'right is might', right? They must be doing something right to be mighty, right? So, if they are evened out, then one sees how the 'not right' and definitely 'not might' are greater then them.

I keep scratching my head trying to think of a better song then that, while listening to ac/dc. And I just can not.

But, I have to admit. All of my friends are so incredibly patient. They have had to keep their mouths shut for so very long. They have kept their security, and that, at extraordinary pain, difficulty. While the ones they have had to keep their mouths shut over have had zero security to them. Completely naked for what they are. They think they are secure because no one has called them on anything.

But... they have been closely surveilled all along. And they have never had any sort of security.

They just think they have. And surveilling people like that? Is there anything more painful on the face of the earth?

To see what that do, to see what they say... to really hate it. But to not say anything.

That, to me, that is real security.

Even if it does not make sense to most other readers.

AndrewMay 21, 2015 4:47 AM

It's all bullshit, nobody wants to patch vulnerabilities at OS and applications level because they want to maintain systems open.

The reason is that at firewall / router level things CAN be secured. On most systems, every OUT request should be restricted to browser + email client + messenger, nothing else should go out.

Its all brainwashing with fancy graphics.

Terry ClothMay 21, 2015 5:04 AM

WINE is not the emulator
@Figureitout, @Vulnerability Researcher:

Note that the article's ``WINE'' is the Worldwide Intelligence Network Environment, a database of ``representative field data about the cyber threat landscape''.
So it seems to be, in fact, real-world data, with an abysmally-chosen acronym.

Tudor DumitrasMay 21, 2015 4:40 PM

I am one of the authors of this paper. I wanted to thank you all for your comments; I am generally interested in war stories about patching (or the lack thereof). In the paper we propose some ideas for mitigating some of these problems, but I would like to understand better what makes patching hard in practice. Now that we have a comprehensive set of measurements, we may be able to run some "what-if" scenarios and come up with better solutions. Drop me a note if you'd like to chat about this; my contact information is on the Web page linked from Bruce's post.

I want to clarify that WINE indeed stands for "Worldwide Intelligence Network Environment" (I did not choose the name). WINE is a data analytics platform with data collected on end hosts (not through network scanning). This platform allowed us to measure how quickly vulnerabilities were patched on ~8 million hosts worldwide. We compared the patch deployment process for 10 client-side applications, including both closed-source and open-source programs.

ZenzeroMay 21, 2015 7:33 PM

@ Tudor Dumitras

Thanks for the article, previous company had that that issue with patching but the current one has a completely different one /sigh

Please keep us updated as you gather more info

rgaffMay 21, 2015 8:18 PM

The biggest real world problem against security also affects patching: that is, people gotta eat, so they're off working on the next product or customer or whatever rather than fixing existing ones... Unless they have to, of course, and that's expensive, and can cause bankruptcy if you spend too much time at it, and it helps the issue even less to go bankrupt!

FigureitoutMay 21, 2015 9:26 PM

Terry Cloth
--Goddamnit Terry!--You're right...guess who didn't read more than the blurp...Before long I'll be ranting like a blow hard how alcoholic security researchers name their tools after drinks!

Tudor Dumitras
--The problem is much bigger than can be fixed w/o painful redesigns and overhauls, that's what we're saying. There's exploits out there that can exchange binaries en route, the patching site needs to be secure, and the patches could break previous applications which they do again and again and again...

I have concerns of a patch environment being the way for attackers to "patch" your system w/ malware.

I do like "apt-get" though b/c it makes things very easy...maybe too easy...

Vulnerability Researcher
--Few things: 1) Do you understand what crippling anxiety feels like? I haven't had a panic attack in years, and I intend to keep it that way.
2) Don't expect good intel when your target knows something's up, and then the rookie-brigade marches in after the contracted pro's came in w/ legal carte-blanche and a good approach and OPSEC (which couldn't be sustained though so I caught a few of them too), infected all my PC's, set up shop in the neighborhood, and followed me once over 600 miles.
3) Tell your asshole friends to open their mouth all they want, I don't care.
4) If you wish to speak, put out an email to blabber on about OT things.
5) Get too arrogant, someone will breach you. Nothing worse then not knowing you're owned.
6) Tell the aliens I said 'hi' too.

anonymous cowardMay 21, 2015 10:19 PM

Ops, bucked up the wrong tree. I wonder who what are they incestigating?

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of Resilient, an IBM Company.