Shadow Brokers Releases the Rest of Their NSA Hacking Tools

Last August, an unknown group called the Shadow Brokers released a bunch of NSA tools to the public. The common guesses were that the tools were discovered on an external staging server, and that the hack and release was the work of the Russians (back then, that wasn’t controversial). This was me:

Okay, so let’s think about the game theory here. Some group stole all of this data in 2013 and kept it secret for three years. Now they want the world to know it was stolen. Which governments might behave this way? The obvious list is short: China and Russia. Were I betting, I would bet Russia, and that it’s a signal to the Obama Administration: “Before you even think of sanctioning us for the DNC hack, know where we’ve been and what we can do to you.”

They published a second, encrypted, file. My speculation:

They claim to be auctioning off the rest of the data to the highest bidder. I think that’s PR nonsense. More likely, that second file is random nonsense, and this is all we’re going to get. It’s a lot, though.

I was wrong. On November 1, the Shadow Brokers released some more documents, and two days ago they released the key to that original encrypted archive:

EQGRP-Auction-Files is CrDj”(;Va.*NdlnzB9M?@K2)#>deB7mN

I don’t think their statement is worth reading for content. I still believe the Russia are more likely to be the perpetrator than China.

There’s not much yet on the contents of this dump of Top Secret NSA hacking tools, but it can’t be a fun weekend at Ft. Meade. I’m sure that by now they have enough information to know exactly where and when the data got stolen, and maybe even detailed information on who did it. My guess is that we’ll never see that information, though.

EDITED TO ADD (4/11): Seems like there’s not a lot here.

Tags: , , , ,

Posted on April 10, 2017 at 5:51 AM • 63 Comments

Comments

Who? • April 10, 2017 6:14 AM

I only hope the full disclosure of these attack tools and software/firmware implants will help developers fix the remaining bugs and make the world a bit safer.

ATN • April 10, 2017 7:49 AM

will help developers fix the remaining bugs

You forgot to say, “for free”. Everybody should work “for free” in the computing industry.

But what I don’t understand is why the fixes do not exist already.
Can you be a highly costy gouvernment agency and keep a list of single day exploit on a computer which is not protected against those exploits?

Also, how a society can accept computers in their police and their justice departments not to be protected against complete remote control? Aren’t there already organisations offering selected information wipe-out on any of those computers and their backups (for a fee)?

Beamboom • April 10, 2017 7:55 AM

That statement, though… What an ugly, racist slur of a statement, looks more than anything like mentally unstable ramblings. If this indeed is (on directions) from a government, why wrap it up in #### like that? What’s the strategy behind it?

Eliza • April 10, 2017 8:06 AM

Mangling the text like that (probaly with the help of a translation-bot) may be to avoid traces of the author’s own vocabulary/speech patterns

MrC • April 10, 2017 8:13 AM

@ Beamboom

It’s a message to Mr. Trump that says, “We, the people who got you elected, are displeased with what you just did. (Continue to defy us at your peril.)” The clever bit is that the fundamental meaning remains the same regardless of whether you understand the message is coming from the Kremlin.

gl • April 10, 2017 8:30 AM

To briefly answer ATN’s three questions:
1. Yes, if someone makes a mistake
2. You ask “How can society accept…” That means “look in the mirror and ask yourself….” But regardless, protecting yourself against intrusion is very, very difficult.
3. Sort of, but even if the answer were “Yes”, that’s not a solution. (See #2)

More generally, ATN’s comment made me wonder if there’s a “Computer Security 101” brief, or blog entry, or whitepaper, or the like. Not something like the Wikipedia page, which is exactly what you would expect (and want) from an encyclopedia entry. I’m thinking more along the lines of “why computer security is so hard”, to provide context so that questions like these are more refined before they’re asked.

I did a little poking, and the best I could find was
Internal to schneier.com: https://www.schneier.com/essays/archives/1999/11/why_computers_are_in.html
External: https://www.cs.utexas.edu/~byoung/cs361/lecture2.pdf

But I’m not sure those really answer the mail, so to speak. I wonder if it’s worth the effort to put something like that together?

Anon • April 10, 2017 9:39 AM

I can’t decide if ATN was serious about people working for free? Why should we work for free? Providing security fixes for free, sure. Working for free, no way!

Is this in retaliation for the strikes, or just coincidental?

Who? • April 10, 2017 10:29 AM

It is not a matter of working for free or not.

Some of us are writing software either open source, free or commercial. Those of us that work on open source or free software do it for different reasons. Mines? I love what I do, I want to give something useful to the world as a gift, and I am developing software that is very useful for my own goals so my work and —even more important— the work of other members of the project are making my life much easier by writing software that is better than most commercial software products available. Others are working for corporations, developing commercial software, writing firmware that runs on computers and other appliances, and getting paid for that work.

I do not care on what branch of the software ecosystem (either “free” or “paid”) developers of the affected products are. A bug is something that must be fixed. A bug is our responsibility —when a bug is found it must be fixed, period.

Anura • April 10, 2017 10:57 AM

@Who?

First off, let’s understand that no one is a slave. If you use software, it’s your responsibility to understand the warranty – with open source, that is generally none. The point of open source is that you are not dependent on anyone else to fix it, but unless you are paying someone specifically to fix something for you then no one is under any obligation to fix any bugs whatsoever, security or otherwise.

My Info • April 10, 2017 11:22 AM

Government bureaucracy: numerous legal technicalities.

  1. If these hacking tools were classified, it would technically be all but impossible to use them on an unclassified computer network without having already violated federal law on the disclosure of classified information. This would all but defeat the purpose of maintaining and using an arsenal of classified hacking tools in the first place.
  2. If the tools were created in the direct employ of the United States government, then they are not subject to claims of copyright.
  3. Any U.S. government claims of “trade secret” violation would sound extremely specious in a court of law at this point: classification is really the only proper vehicle for the government to keep a secret.

In a practical sense, conclusion #1 is a consequence of simple reality: we are deluding ourselves if we think we can use our most sophisticated “hacking tools” against our most sophisticated enemies without effectively disclosing those methods to the enemy. Conclusions #2 and #3 are basic concepts of trade law. Patent law is irrelevant. We cannot expect patent law to prevent hostile nation-state use of patented inventions.

@Anura

First off, let’s understand that no one is a slave. If you use software, it’s your responsibility to understand the warranty – with open source, that is generally none. The point of open source is that you are not dependent on anyone else to fix it, but unless you are paying someone specifically to fix something for you then no one is under any obligation to fix any bugs whatsoever, security or otherwise.

Absolutely correct. If you want it done right, do it yourself. That is really the point of open source. With proprietary software, the source code withheld by trade secrecy, and the executables distributed under onerous copyright conditions, you do not even have that option.

Realize, too, that all too many people have an aggressive interest in sabotaging open source for everyone else because they have a competing proprietary solution for sale.

albert • April 10, 2017 11:34 AM

@Anura,

“…If you use software, it’s your responsibility to understand the warranty – with open source, that is generally none….”.

“warranty”? Look it up, then read Microsofts EULA.

Software providers of any kind don’t provide warranties, by any definition of the term.

They absolve themselves of -any- responsibility for -anything-.

Yes, I’m overstating for emphasis.

@Who? is talking about -personal- responsibility. It’s a big difference. Corporate ‘responsibility’ entails keeping the users from abandoning their products.

. .. . .. — ….

Anura • April 10, 2017 12:02 PM

@albert

Not all commercial software comes without warranty – just generally not software directed at the general public unless it simply allows for a refund without requirements to fix bugs. Generally speaking, whenever software needs to be regulated (e.g. flight software) any software is going to require a warranty.

And no, there is no personal responsibility to fix that software either. If you aren’t getting paid, then you have no personal responsibility to maintain any of the software you wrote. If you wrote software and it turned out there was a bug that is causing havoc to those using it then you can lie back, decide not to worry about it, and sleep comfortably knowing that unless you advertised otherwise, you have absolutely no responsibility to take any corrective action whatsoever. As long as you provided the source code, they can fix it themselves or pay someone else to fix it.

supersaurus • April 10, 2017 1:50 PM

@Who?

“…A bug is our responsibility —when a bug is found it must be fixed, period…”

oh really? are you assuming seeing a bug is sufficient information to enable fixing it? I worked on a large system analysis tool that would freeze up about every 6 months to a year (mind you that window might encompass thousands of new executions of the tool, it isn’t one of those “run forever” things, they have their own set of problems). you would then restart it on the same machine etc etc etc and it would run fine thousands more times. if you owned the company how much would you expend to fix a bug like that? before answering bear in mind this was hundreds of thousands of lines of multithreaded code, which means the chance of reproducing the exact circumstances that made it manifest was approximately zero. given that, consider what a debugger does to the execution order of instructions in multithreaded code.

not enough? then consider that that product had easily thousands of bug reports, many of which were either not verified or could not be verified. which is more important, a bug that shows up half a dozen times in the life of the product or a bug that is merely annoying but has been reported by a dozen customers and reproduced by a developer?

oh, right, if you have one developer per bug then no need to prioritize. don’t forget to add another 25% or so developers to fix the bugs introduced when the previous bugs are “fixed” and the cascade that follows.

and that particular bug showed up on machines that had only two processors…think how much more enjoyment one can have running on a machine with 8 logical processors, or even better, GPU code where thousands of processors may be involved.

so kindly don’t generalize to include me when defining “our responsibility”.

ab praeceptis • April 10, 2017 1:54 PM

Bruce Schneier

I still believe the Russia are more likely to be the perpetrator than China.

And I’m amazed how easily and quickly you paint Russia (or China) as guilty.

Well noted not so much because it’s utterly unfair; both Russia and China are quite used to arbitrary accusations coming from the us of a.

No, it amazes me because you, after all not some nobody but a widely recognized security expert, seem to consider the security in all its variations, e.g. OpSec, of your countries most sensitive and presumably professional organisations as so utterly poor, that “it was [insert favourite evil enemy country]” seems to look quite natural and credible.

And yes, I think that you are right. (Which is bad for your country but very good for this world).

I will stay away from it as it would risk to turn into an ugly political thing, but it should be rather obvious:

How exactly does the us of a assume all their accusations and smearing to have any desirable effect anyway if it’s clear that the Russians and Chinese have wires deep into washingtons “brain”? Isn’t that akin to the mob boss making lots of noise against fbi professionals who know even intimate details about what’s going on in his mansion?

ab praeceptis • April 10, 2017 2:06 PM

supersaurus

When you don’t have the needed capabilities and knowledge to do a complex then you should stay away from it or look for help/advise. The people handling venomous snakes in zoos will certainly agree with my statement.

@Who

Well spoken. But there’s an ugly but: The working time of developers, at least in companies, is rarely theirs to manage. And those who do manage it are more often than not clueless profit driven halfbrains.

But there is a way more important point to be made:

Even in most foss projects there is a usually not small zoo of bugs. Are they all stupid and/or lacking good intentions? I don’t think so. It just seems that humans and the common working processes are bound to create bug zoos.

The solution? Design properly before hacking. Work with proper tools and an adequate language. Be sure to properly verify at least the core and/or the sensitive parts of your project.

In other words: Don’t kill living bugs but program with “condoms” so as to keep the unborn.

Ross Snider • April 10, 2017 2:27 PM

Wow their statement was incredibly potent and speaks directly to Donald Trump’s base.

Also, they do bring up a good point regarding the Russia speculation:

“For peoples still being confused about TheShadowBrokers and Russia. If theshadowbrokers being Russian don’t you think we’d be in all those U.S. government reports on Russian hacking?”

Dirk Praet • April 10, 2017 3:20 PM

One of the preliminary takeaways is that despite being all up in arms about ubiquitous Russian spying even on arguably valid political targets like the DNC, very few US MSM can be bothered being outraged (or even reporting) about their own IC installing implants on about 910 different servers, most of them civil infrastructure in allied countries. And on top of that being incapable of keeping it secret. Not to mention spying on universities probably having nothing to do with a vested interest in R&D and IP, one of these other very persistent US espionage myths.

I find the hypocrisy disturbing, and I wish more people – also on this forum – would finally start focusing on what’s in those documents than on who was or was not behind their leaking.

supersaurus • April 10, 2017 5:17 PM

@ab praeceptis

I didn’t understand your comment addressed to me in re “…do a complex…”, so can’t reply.

but I can comment on this:

“The solution? Design properly before hacking. Work with proper tools and an adequate language. Be sure to properly verify at least the core and/or the sensitive parts of your project.”

I assume you mean if a program isn’t perfect then by definition it wasn’t done according to your guidance?

and where is the boundary between “core and/or sensitive parts” and the rest of the code? will my editor highlight it? I find it difficult to draw the line between “must work ‘properly'” and “we don’t care” sections of code…in my experience there had to be a reason to write a particular piece of code, in other words there was no “don’t care” code; and considering the cost of development it needs to be a pretty good reason.

just out of curiosity, some questions:

1) have you ever worked on commercial software, i.e. paid development of a product with a ship date that is intended to generate profit and is actually sold?

2) what approximate kloc was this project(s)? (yes, I’m well aware that’s a blunt instrument, but it does tell you something about testing difficulty, build system complexity, etc; I doubt you would argue that a program built from a million lines of code and thousands of individual files (you can pick the language) would not be significantly more difficult to maintain than a 500 line utility)

3) if you answered “yes” to question 1), what was the lifetime of the product, i.e. for how many years was it sold and hence maintained?

4) if you answered “yes” to question 1), did the product run on multiple operating systems? if yes, which?

ab praeceptis • April 10, 2017 5:44 PM

supersaurus

Apologies; there should have been “project” after “complex”.

As for the rest: Nice try.

For a start, I’m working in software development (and later design) since about Compaq offered the first PC clone. The projects (and clients) were anywhere between small and very large; I guess quite many colleagues have a similar background.
So, your implied “I have more real world experience and in larger projects” is a rather blunt attempt.

As for the matter itself: Oh, please, feel absolutely free to contradict both leading experts as well as massive real world experience. Because pretty much everything we have learned clearly points towards proper spec and design followed by proper implementation (using languages that lend themselves to verification).

Moreover, we have learned that repairing existing code bases, particularly larger ones is pretty much a lottery in which killing one bug creating another one is quite common.

There is a reason for the aeronautical industry, fintech, railway systems, etc., i.e. fields where bugs can lead to many lost lives (or billions of $) are preferring Ocaml, Eiffel, Ada rather than C[++] or java and there is also a reason for tools like B (spec, modelling) being often used in those fields.

One reason, btw. is the “cost rocket”, i.e. the dramatic increase in cost of bugs being found late. A bug found in a shipped product typically costs 100 to 1000 times more than avoiding or catching it early during development.

I’m afraid you picked the wrong guy for your game. Have a nice day.

supersaurus • April 10, 2017 6:45 PM

@ab praeceptis

I didn’t intend to imply anything since obviously I have no way to know about you except by asking questions.

I guess I did pick the “wrong guy”, since you avoided answering perfectly civilized questions and instead resort to argument from authority rather than information.

I do agree with your paragraph on the “cost rocket”, but that is not new information nor does it answer the questions. Same for the cost and difficulty of repair, also not germane and not new information for me.

I wasn’t aware this was a game, but apparently the first rule is “rudeness can substitute for substance”. are there other rules? pray, enlighten me, maybe your jotting would make more sense if I knew them.

ab praeceptis • April 10, 2017 7:02 PM

supersaurus

Stop lamenting. Your questions have no weight in the facts. Ocaml, for instance, allowing for less buggy software is completely independent of my experience, the kind of projects I worked on, etc.
Similarly, my curriculum vitae has no influence whatsoever on java programs being easy or difficult to debug or on using at least ESC or not.

And you repeat your personal games right again by arbitrarily calling me rude. Id suggest to rather offer arguments demonstrating your position in the matter as tenable. You chose not to – that’s not my responsibility.

In case you are interested in the matter I’m ready for a constructive discussion. Your choice.

supersaurus • April 10, 2017 7:34 PM

@ab praeceptis

“Your questions have no weight in the facts.”

???

well, I’m sorry if I hurt your delicate little feelers and provoked your ire. we are wasting column inches and probably trying our host’s patience. since I respect our host and enjoy both the blog and intelligent discussion in the comments I will now cease beating a dead horse.

My Info • April 10, 2017 7:43 PM

@albert,Anura

After you’ve charged good money for proprietary software and locked your customers’ data in to it in such a way as to unreasonably escalate their expenses of moving away to a competitor’s product, you are hard pressed to “disclaim” the “implied warranty of fitness for a particular purpose” in the fine print — I mean, that’s what you advertised your software as, what your customers thought they were paying for, and what you told them they would be able to use it for when you took their money for it.

Open up the package, read the fine print: “I’ve been hornswoggled! The software I bought isn’t fit for anything! I have no rights or recourse!”

If you buy a spade, and it’s so flimsy that the handle breaks off the blade the first time you attempt to turn over a clod of loose soil with it, you’d probably take the spade back where you bought it from and demand your money back.

If you bought a 10-year multi-million-dollar contract for such spades, and they all failed in like manner, you bet your boots you’d be suing the vendor in a court of law, because you legitimately thought you were buying digging implements, not a suit of cards.

Speculation • April 10, 2017 8:01 PM

Years ago, an oil rig fire was extinguished with dynamite (or some similarly powerful explosive?) at the base of the fire. Seems counterintuitive, but it was effective.

A wild thought struck me as I read this article. What if the shadow brokers are the dynamite?… to throw off the trail or extinguish worse leaks?

4 year old code likely stolen from a staging server, once ransomed, and now revealed by the midst of a mere tantrum over politics? The whole thing is irrational. It smells like a red herring, or a feint in an arm wrestling contest designed to drain the opponent’s energy.

Meh. Maybe I’ll have another cocktail and think again.

Nick P • April 10, 2017 8:56 PM

@ supersaurus

Let me weight in on this. The simpler version of the tangent is that programming most of this stuff in memory-safe languages with solid crypto, remoting, and OS’s would prevent vast majority of hacks. May or may not stop nation states who just find another 0-day or eventually invent new classes of them. Yet, almost everything hit in leaks or real-world compromises involved exploiting a low-level bug in software. Plus, the NSA has available to them fabs and budget for processors using hardware modifications that make most 0-days impossible with less loss of performance. At least one, Cambridge’s CHERI, already runs FreeBSD. There’s also a ton of OS’s and middleware, both FOSS (eg OpenBSD, DJB’s) and proprietary (eg separation kernels, SNS Server, HYDRA), that stop all kinds of attacks. They could use more but don’t. Part of that is corrupt politics over COTS acquisitions, though. And the NSA itself killing off much of the high-assurance security industry earlier on. 😉

My Info • April 10, 2017 9:23 PM

@Speculation,Nick P

Right. Let’s just throw OpenBSD and DJB in the same bathtub and watch what happens.

Right now it’s a bar fight with a gang of angry abogadas. You can’t even talk about your case privately with an abogada and there’s absolutely no attorney-client privilege in this district because the old avogado is furious that you’re consorting with his granddaughter….

My Info • April 10, 2017 9:28 PM

… and that’s what happens when the NSA/CIA government starts holding people incommunicado without abogado.

Ratio • April 10, 2017 11:02 PM

@supersaurus,

I worked on a large system analysis tool that would freeze up about every 6 months to a year […]. if you owned the company how much would you expend to fix a bug like that? before answering bear in mind this was hundreds of thousands of lines of multithreaded code, which means the chance of reproducing the exact circumstances that made it manifest was approximately zero.

My condolences.

Do you think that the people who created such an intractable system in the first place get to use this excuse?

Who? • April 11, 2017 2:37 AM

@ Ratio

Do you think that the people who created such an intractable system in the first place get to use this excuse?

Exactly. This one is the problem. FOSS software projects usually start with a clean and high-quality code base, in the case of OpenBSD it is the Berkeley Software Distribution, as taken from an earlier FreeBSD repository through NetBSD. These projects improve the base over the years, share the changes and source code with the community so thousand of users can provide feedback, and make uncompromising design decisions1 changing anything they feel is wrong.

As a result we get clean and well maintained software projects. Do these projects have bugs? Of course, thousand of bugs! But these bugs can be usually tracked and fixed in just a few hours.

1 On the other hand, commercial software products are either provided without warranty (see @albert comments about Microsoft EULA) or are the result of poor design decision taken by customers paying lots of money for the right to use these products and imposing their “requirements” to developers. Is this one the right development model?

supersaurus • April 11, 2017 9:14 AM

@Nick P

thank you for your thoughts.

“…memory safe languages…

this is made difficult by the sheer size of the COTS code base (including OS) written in c/c++. good coding practices in c++, e.g. RAII, make writing memory safe code easy, but many shops don’t even do that let alone run available analysis tools. multithreading makes runtime behavior an order of magnitude more difficult to understand and test (if it can be done at all), and the now common appearance of multi-cpu hardware is exposing bugs that never showed up in multithreaded code running on a single processor.

an amusing thing about that twice a year bug I mentioned is part of the control flow in the code was originally based upon M$ sample code (I’m not making this up) using “events” rather than mutex et al. afaik nobody in their right mind would do that (use events) because of the debugging difficulty, but that choice was made years before I got there. why didn’t we change it? fear of other consequences…that bug popped up so seldom that it wasn’t worth the risk.

in re crypto, the number of shops who believe in “rolling our own” makes it easy to see why there are crypto issues.

and now people are talking about self-driving cars. hardly any software involved, and all simple to understand, right? right? maybe I’ll be dead before they are common out here in the sticks :).

Bruce Schneier • April 11, 2017 9:28 AM

@ab praeceptis

“And I’m amazed how easily and quickly you paint Russia (or China) as guilty.”

Mostly it’s Occam’s Razor. I am assuming that this was a sophisticated operation, and 1) the work of a government with a sophisticated cyber-intelligence operation. I am also assuming that it is 2) a country whose intelligence agency is working against the United States and 3) would be willing to embarrass the United States. (That third part ruled out Israel.) Aside from Russia and China, I can’t think of any other countries that would fit the bill. And in 2016, Russia more than China.

supersaurus • April 11, 2017 9:40 AM

@ Ratio

“…Do you think that the people who created such an intractable system in the first place get to use this excuse?…”

I’m sure what happened there is management said something like “we need a system that does X and we need to ship yesterday” and then the developers said “you point me, and I’ll march”. considering what this particular system did I doubt whether any of the original developers had any experience with something similar, and it didn’t really show up for years. since you couldn’t reproduce it, and it happened so rarely nobody had incentive to take the risk.

@ Who

100% of all software projects eventually trace back to a “clean and high-quality code base”, namely a single empty file, but once you have code that executes you are dependent upon possibly millions of lines of code in the operating system and runtime libraries. all that system code also started with a single empty file, but by the time you have enough system code to actually do something you are out of the realm of code bases that can be reviewed by an individual in a commercially viable time frame.

“Do these projects have bugs? Of course, thousand of bugs! But these bugs can be usually tracked and fixed in just a few hours.”

I agree with the “thousand of bugs!” assertion (although rather optimistic in magnitude), but otherwise a rather broad generalization. do you have a track record fixing OS bugs?

Bugfinder • April 11, 2017 10:15 AM

Nice discussion about bugs… If you want to buy bug free software, because failure would be life threatening, you will obviously need a defined hardware to run this software, failure of the power supply because the UPS left a mains spike pass through is a bug and would kill.
Nobody will guaranty running on an uncontrolled operating system, on a random PC, with random video card and random network card with random drivers. All those will be fixed versions, maintained by the provider of your software, so the system will involve maintenance contract linked to guaranty. Remote maintenance may only be provided with high level encryption those export is restricted.

Evan • April 11, 2017 3:30 PM

@ab praeceptis @supersaurus

You guys both have some strong talking points but ultimately supersaurus position is more pragmatic and realistic. The idea that using the right tools makes for a good project is in my experience pretty off basis. The idea any language can be memory safe is flat out wrong, there is no such thing as memory safety in computing, period. Whatever tools you’re using are running on top of a platform that is inherently limited by real world constraints.

The irony of you touting those languages as being safe while criticizing Java is that those same kind of statements are Java’s claim to fame. Remember, it runs in a VM, it’s memory safe. Oh wait, the VM gets memory from the OS and OS issues memory based on the hardware which has a finite and well known limit. If you think those languages are inherently safe, you’re completely wrong. If they become market leaders I guarantee you they’ll end up having tons of security exploits exposed like all the other market leaders.

People tend to ignore economic theory heavily when discussing topics regarding technology tooling. For example, Apple has retained a reputation among consumers as being far more secure than Windows. How much of that do you think is due to OS X’s historically low market share, rather than actual technical differences make OS X more difficult to attack? Yes, I would agree a Unix based OS is most likely more secure than Windows but do you think it was that or Windows 97% market share driving the higher rate of vulnerability discovery in Windows? Obviously it was the market share and not the technology.

Another point where you’re off basis… You used a bunch of industries that are technically incompetent as the basis of evidence that these languages are inherently more secure. From what I’ve observed in the software industry, it is highly likely that the average engineer working at a company who’s main product isn’t software is far less skilled than the average engineer working at a company who’s primary offerings are in software. That is also supported by economic theory. Where do you think the software development talent is going? To some random contracting firm? To an airline? To Google? Again, this should be obvious. If those languages were so great for security, they would be sponsored by a major company and be a major part of their product offerings, they’re not. It’s not a coincidence.

ab praeceptis • April 11, 2017 3:43 PM

Who?

Really? Much of foss, incl. what has been set free by companies, never was properly designed.

At the same time we can’t but notice that one of the biggest players in proper software design and implementation is microsoft.

And there is another factor, albeit a somewhat smelly one that is often overlooked: The most important factor is knowledge and capability, no matter whether in foss or in ccss. But it just so happens that the average level of professional capacity is dimensionally higher than in foss projects. Of course, after all the companies pay and can select.

I personally don’t care much and am not religious in either direction. For me there is properly engineered software and crap, no matter who did it.

ordinary average guy • April 11, 2017 3:48 PM

“…and that the hack and release was the work of the Russians (back then, that wasn’t controversial).”

Agreed. It wasn’t nearly as “controversial” until our current crop of lazy propagandists decided to start their sophomoric attempt to resurrect Joseph McCarthy (likely as a spastic response to a perceived loss of control due to neither of the establishment candidates winning the presidential election).

…we really don’t even need to entertain such silly notions as the reason neither establishment candidate won was due to both being completely out of touch with / acting in complete disregard to / the best interests (esp., financial) of the majority of their constituents. Naaaah… It was totally because of the Russians.

It’s hard to put too fine a point on the fact that the organizations promoting the Russian explanation are professional propagandists with a long/current history of lying to the public. Here’s some perspective as to why the Russian explanation should be treated with extreme skepticism:

The Increasingly Unhinged Russia Rhetoric Comes From a Long-Standing U.S. Playbook
https://theintercept.com/2017/02/23/the-increasingly-unhinged-russia-rhetoric-comes-from-a-long-standing-u-s-playbook/

That being said, I think it would be naive to assume that the Russian (like the US) don’t attempt this sort of thing as normal policy. So to me, the most relevant question becomes, why is our political establishment making such a show of it now?

Also, if it is the intention of foreign governments to provide the US public/law enforcement with legitimate evidence of official malfeasance (e.g., DNC hacks), I couldn’t more fully support that effort. I do NOT care in the slightest what their motivation may be. We must weed out and prosecute the bad actors in our institutions if we ever hope to restore any semblance of integrity in those institutions.

ab praeceptis • April 11, 2017 4:16 PM

Bruce Schneier

OK, I see how you arrive there but a) large != capable. The french, for instance, have plenty reasons to hate the us of a and they certainly have the necessary education system and resources. b) and more importantly, how about the us of a itself or some insiders?

I think we should be careful with attribution because it creates serious harm and that tends to have consequences (e.g. demands for compensation).

In fact, right now, the base of objective facts we have do point to Russia being a victim (cia painting them as hackers) while there is plenty of evidence against the us of a.

What will you say if one day Russia demands 20 bln $ in compensation for all the baseless wanton libel against it?

ab praeceptis • April 11, 2017 5:48 PM

All

What are we talking about? Unless we know that the discussion may be entertaining but it’s largely baseless (and hence meaningless).

Similarly, diverse attributions and discussion about those may be entertaining as may be discussions about diverse “opinions” about safe and reliable software.

BUT: The malaise of everyone and his little sister uttering opinions have been a major factor in the ugly and grave situation we’re in in the first place.

We don’t need opinions or left or right or trump or clinton. What we need is secure networks, reasonable and trustworthy levels of privacy and sound software. The “democratic western” blabla approach has failed painfully obviously. Isn’t it about time to finally go in a professional way about it?

For a start, let us forget about all those funny idealogies and rather start to think clearly.

Society, for example, can – and should – define what we want. It can not, however, define, how that is achieved. That is up to the engineers and it’s about time that they actually do that.
We need properly designed and specified software that is also properly implemented.

We have the necessary tools (bar one, namely enough well educated engineers). Granted, those tools often are still somewhat uncomfortable, but anyway we do have them.

So, let us finally apply and use both the right approach and the right tools.

That – and only that – will bring us closer toward our urgent goals.

Max • April 11, 2017 5:50 PM

Everyone is oddly happy that the brief but intense love affair between Donald Trump and Russian government is over. Things can go back to business as usual. What a relief!

supersaurus • April 11, 2017 8:32 PM

@ab praeceptis

“…Society, for example, can – and should – define what we want. It can not, however, define, how that is achieved. That is up to the engineers and it’s about time that they actually do that…”

I never worked for nor ever heard of a for-profit software company who left scheduling up the engineers. this could happen for a while in a one-horse outfit, but that size operation isn’t significant in the big world anyway. my managers always had the odd idea that since they signed the paychecks they could set the schedule, and they did: full speed ahead and damn the torpedoes.

“…We have the necessary tools (bar one, namely enough well educated engineers)…”

commercial for-profit software sold into a market big enough to support more than one player will be driven by the pressure to ship unless and until the vendors are held to account by the courts for consequential damages. good engineers do the best they can in the time allotted after they push back as far as possible (which is usually “not very”), but finally engineering doesn’t control the schedule.

my anecdote goes like this, and I’ll bet any developer here working on commercial software can tell a similar story: 1) we meet with management and foolishly admit that we might actually make the ship date; 2) management pulls the ship date in by 3 weeks with no change to deliverables.

you can argue all day about the tension between quality and ship date, but however you slice that I don’t believe engineers are the principal cause of poor commercial software quality; certainly not in the shops where I worked.

ab praeceptis • April 11, 2017 8:38 PM

supersaurus

I don’t see against whom or what you are arguing but it’s certainly not to do with what I wrote. It might be helpful to understand before arguing against something.

Nick P • April 11, 2017 9:14 PM

@ supersaurus

“this is made difficult by the sheer size of the COTS code base (including OS) written in c/c++”

This is true when we’re talking about weird, legacy codebase. However, many apps can be improved just by porting off complex components onto simplified ones that are already secure or could be. Others might use a tool like Softbound+CETS to automatically improve safety of the apps. Likewise, there’s libraries like SaferCPlusPlus that one might add a piece at a time. You’re right that few try.

“using “events” rather than mutex et al. afaik nobody in their right mind would do that (use events) because of the debugging difficulty”

You think it’s funny they used events but I don’t. [I do think it’s funny they got burned copy and pasting sample code lol.] I personally have gone back and forth over this topic for a long time. I don’t know whose right or what to believe on event-based vs safe multi-threading or shared-somethign vs shared-nothing. The highest-availability machines, NonStop, were shared-nothing along with one of highest-reliability languages, Erlang. Easiest to use and verify for a long time were Ada Ravenscar and Eiffel Scoop that are more careful sharing. The local networks maintained integrity/availability with synchronous protocols where they started breaking CAP-style on geo-distributed, data centers. Many got better luck with asychronous protocols whose state got synchronized over time or something. I just throw my hands up on The Ultimate Answer then pick something that worked in whatever was most similar to current thing I’m exploring. Pragmatism.

Far as Microsoft, though, they’re back at that asynchronous strategy with a tool that already improved reliability of their complex, USB stack.

“in re crypto, the number of shops who believe in “rolling our own” makes it easy to see why there are crypto issues”

You’re still thinking about Microsoft. It’s OK to admit it. They were idiots with crypto. So were much of the industry but Microsoft’s stuff got pounded. They’re sort of redeeming themselves with miTLS.

@ Evan

“The irony of you touting those languages as being safe while criticizing Java is that those same kind of statements are Java’s claim to fame. Remember, it runs in a VM, it’s memory safe.”

That’s way off. Java’s safety was marketing BS. Look up Modula-2, Modula-3, or Oberon to see a memory-safe language. They don’t have a huge runtime. They don’t have interpreters. They don’t have residual unsafety in the semantics. The standard library can be written in same, memory-safe language. Even whole OS. When unsafe is needed, esp low-level manipulation of hardware, it’s done in modules labeled UNSAFE or SYSTEM that tell you all bets are off. They typically wrap those in type-safe interfaces w/ checks of arguments or whatever else.

Java is so loaded with complexity and C (or C++… whichever) that it was piles of vulnerabilities waiting to happen. They happened. The others had many fewer in terms of language or runtime because the language were straight-forward constructs to map to machine instructions with checks or GC.

“Apple has retained a reputation among consumers as being far more secure than Windows. How much of that do you think is due to OS X’s historically low market share”

All of it. They’re getting better, though, with sandboxes and such. Strongest on mobile outside cryptophones or expensive enterprise packages.

“but do you think it was that or Windows 97% market share driving the higher rate of vulnerability discovery in Windows?”

That’s a different question. It’s provably not the case for Windows vs quality BSD’s where Microsoft was throwing shit together, purposely supporting buggy 3rd party software, and reinventing things like crypto or standard protocols instead of using proven ones. Microsoft’s mantra was “Ship First, Fix Later for Money.” Like he said when running their security program. 😉 You could say similar things about Linux ecosystem, too, where extra market share found a lot of vulnerabilities. Whereas, they’re not finding much in OpenBSD no matter how popular it is since their culture is opposite of Microsoft: fix, remove, or debug more code than you add in terms of features.

Market share only explains so much. The priorities and methods of those developing software matter more. Also, we called the safety failures top 10 vulnerabilities for a reason: almost all of them would’ve been stopped with memory-safe languages and/or input validation checks. That easy.

“Where do you think the software development talent is going? To some random contracting firm? To an airline? To Google? ”

All over the place. Mostly financial sector as they find tech sector is a bunch of BS. Lots of people switching majors to get more lucrative jobs with less age discrimination over time. The rest are a mix with Google etc getting lots of them. Yet, most of the robust tooling I see is made by smaller firms or teams scattered all over the industry. The thing that’s consistent is their management wants it done right & team picks best tools/methods that help them do that. At Altran/Praxis, that’s using Z specs w/ SPARK Ada. At NASA JPL, it’s C w/ strict, coding guidelines + static analysis. At Jane St, it’s Ocaml. At Bank of America, many doing mission-critical apps are using Python with unit tests. All use thorough acceptance tests and code reviews. Believe me when I say it doesn’t take top talent to code Python according to guidelines, look at it carefully, or write tests. Yet, they’re getting it done.

“If those languages were so great for security, they would be sponsored by a major company and be a major part of their product offerings, they’re not.”

What C-level executive or senior person do you know of in a non-technical company that (a) knows security systematically, (b) knows specific languages address these concerns, and (c) is willing to sacrifice a business objective to invent that language on company’s time? I don’t know about any. They mostly ignore security or tooling as an externality. The exceptions were tech companies needing something for highly-reliable systems that knocked out vulnerabilities as side effect. Burroughs B5000 was first with it immune to most vulnerabilities for decades. Unisys still sells it but only language security not hardware checks. IBM did PL/S for their mainframes considering it so important they didn’t want to share it. Likewise, did AS/400’s that virtually never crash in a PL/I subset for microcode called PL/MP. Ericsson needed something to easily make ultra-reliable, telecom gear. That led to Erlang w/ much results in scalable, high-availability systems. The DOD mandate for systematically safe software led to Ada which cut defects in half per studies by Mitre, etc. Praxis created SPARK which proves absence of vulnerabilities without testing.

Past that, the rest were developed by CompSci or independents, a tough sell for most companies, and worked wonders in any company (usually small to medium) that adopted them. One consistent benefit is that they reduce number of bugs from undefined behavior or concurrency that are really hard to track after the fact. Boosts productivity.

ab praeceptis • April 11, 2017 10:28 PM

Nick P

I was bewildered by his post and also didn’t find enough relevant to respond. As you picked it up, though, I’ll add something.

It’s unfortunate that many haven’t been taught about it. But actually it’s quite simple, at least the basic structure. A threat (or whatever variant) is something corresponding to a OS unit; let’s call it “the smallest and simplest form of a process”. Event handling, on the other side, is something corresponding more to the hardware as well as to a reasonable system model.

Let me deviate and elaborate somewhat as I think that’s important.

We have come to see computers as something like factories. Lot’s of material going in, being processed in production lines, and finally lots of results coming out. Only that in our field it’s data in, processing being done, data coming out.

And that was reasonable and pragmatic some decades ago when performance and data throughput were usually the barriers.

Looking from a somewhat wider perspective, though, that’s nonsensical or, more precisely, looking at just 1 kind of processing.

Today, it’s quite different. Today pretty much any system can be regarded as a big fat corresponding event reactors (I’d love to put it better but, alas, my poor english, apologies), most of which require little processing. Just think of lots of network packets arriving just to be written to a disk.
Or think of the gazillions of routers and servers of which even http servers do little processing per event (unless they use bloatache plus php cancer).

Et voila, there it is, the right picture of an event. Events are small but frequent and very often their arrival can’t be calculated. Often they also require a fast reaction.

(Note: This is not directed at you. I’m quite sure you don’t need that lecture. You just happen to have it triggered).

That, btw. also means that multithreading is utterly too big a hammer for most network related jobs.

I have a nice example with my “ssh” server (offering the same ssh offers but a) well designed and built and b) without bloody f*cking ssl/tls):

For the sessions themselves I use threads (Ada tasks). But for the guardian front I don’t; there I use events. They don’t do a whole lot. A connection attempt is made and the other side gets served a challenge (sorry, am not allowed to talk more about that) and only when this gets successfully solved (which to verify also takes just single digit musec) another process starts (or assigns) a thread that then performs roughly what normal ssh does (starting with authentication).

That thingy can cope with hundreds of thousands of connection attempts per second (and such mitigate to a nice degree [D]DOS and other attacks) costing next to nothing per attempt. Classical case for events.

Which finally brings me to one grain of salt I have with Ada: a) task can mean a lot (but is usually implemented as thread) and b) I’d love to have (similar to ‘task’) an ‘event’ type. But, some might say, that can be done anyway. Yes, it can but not with the full power (and verifiability!) of Ada/Spark. sniff.

Ratio • April 11, 2017 10:39 PM

@ab praeceptis,

For a start, let us forget about all those funny idealogies and rather start to think clearly. […] So, let us finally apply and use both the right approach and the right tools. That – and *only* that – will bring us closer toward our urgent goals.

One approach0, one set of tools1; there is no other way.

Also, other people’s positions are funny ideologies, while my own positions are not ideological in any way: they are clearly thought out and based on facts and stuff.

0 The right2 one, of course.

1 You guessed it, the right2 ones.

2 Right is whatever I say is right.

ab praeceptis • April 11, 2017 11:03 PM

Ratio

I presume that was meant to be funny? To avoid misunderstandings:

A telling what are matter of fact the right tools, etc != A trying to establish his personal preferences as rule.

Unlike many here I actually do offer quite many arguments – and not the opinion type.

Note, for example, that I occasionally mention Ocaml, which I do not particularly like, but which objectively does lend itself well to create good quality software, while I rarely even mention, let alone recommend, Modula-3 which I love a lot but which is hardly in a widely usable state, let alone having a bright future.

What I say is based on what has proven to be the right way since millenia. It’s not my invention, nor my preference, nor something I don’t like. It’s simply reality.
Bridges that still stand after many decades (or even centuries) have been designed and built according to the principles of engineering. Same with airplanes that usually don’t fall out of the sky. Same in many other areas.

Feel free to show me a better way or how C can bring us a great future and safety.

supersaurus • April 12, 2017 8:56 AM

@Nick P and others

a clarification, when I mentioned “events” I was talking specifically about win32 events, I’ve no knowledge about other kinds on other platforms. I think I was permanently scarred working on a specific chunk of impossible win32 code that was event based.

@Ratio
Argument from authority makes all else irrelevant ;).

@ab praeceptis

“I don’t see against whom or what you are arguing but it’s certainly not to do with what I wrote. It might be helpful to understand before arguing against something.”

that’s funny, your statement above exactly reflects how I feel when I read some of your ramblings. let me simplify, specifically referring to for-profit software development:

1) engineers don’t set the schedule except by failing to make it
2) for other questions about engineers and scheduling go to item 1)

supersaurus • April 12, 2017 9:05 AM

@Nick P

@ supersaurus

“this is made difficult by the sheer size of the COTS code base (including OS) written in c/c++”

This is true when we’re talking about weird, legacy codebase.

I beg to differ there, I’m sure it is reasonable to guess the codebase of current for profit applications and OS written in c/c++ is well into the hundreds of millions of lines of code. certainly there is a tremendous amount of code that could be improved in various ways, but management generally has no incentive and hence engineering gets no time.

supersaurus • April 12, 2017 9:25 AM

@Nick P

At Bank of America, many doing mission-critical apps are using Python with unit tests.

that’s interesting. I’ve never tried debugging problems in the python bytecode compiler (which btw in CPython is written in c) but I have done so with a java VM. very entertaining, same code ok on one platform, not on another. we used to refer to java as “write once, debug everywhere”.

@ all
yes, sorry, I’m aware that programming languages are akin to religion, but I did find the idea of a bank using python for “mission-critical apps” surprising; probably they don’t suffer from the multiple platform issue because those apps aren’t COTS, but I’m still surprised they got management to accept it.

Sean • April 12, 2017 2:10 PM

I personnally agree that attribution here should be more difficult than the way it sounded. I wouldn’t bet against the US itself either.

Once your spying methods are disclosed, and when facing a world wide islamist threat, what would be a better solution than quite righly misleading everyone about your real undiscovered spying capabilities ?

Ratio • April 13, 2017 8:42 AM

@ab praeceptis,

I presume that was meant to be funny?

Not really.

Unlike many here I actually *do* offer quite many arguments – and not the opinion type. Note, for example, that I occasionally mention Ocaml, which I do not particularly like, but which objectively does lend itself well to create good quality software, while I rarely even mention, let alone recommend, Modula-3 which I love a lot but which is hardly in a widely usable state, let alone having a bright future.

What are the objective criteria one uses to determine if OCaml (or Modula-3 or whatever other programming language) does or doesn’t lend itself well to create good quality software? What is the data that your assertion is based on?

There are objective criteria and data, right? Or is this “not an opinion” actually just another opinion? And if it is, that’s fine, lots of people have opinions. Just don’t say it isn’t an opinion when that’s all it is.

What I say is based on what has proven to be the right way since millenia. […] Bridges that still stand after many decades (or even centuries) have been designed and built according to the principles of engineering. Same with airplanes that usually don’t fall out of the sky. Same in many other areas.

Who could say no to that? (Then again…)

The problem, though, is that different people have different ideas, aka opinions, about what that means in practice. (And that includes the same people coming to different conclusions in different situations. I doubt even you use formal methods for a throwaway script or the odd cron job. If that’s so, why don’t you? And if not, how about when you’re working in a REPL?) So even if everyone agrees on the principle, the question remains “now what?”

ab praeceptis • April 13, 2017 6:01 PM

Ratio

There are many angles from which to answer that. One simple (but funnily reliable) perspective would be empirical observation. I personally take that with a grain of salt, though, as the size of the code bodies are drastically different; it’s very hard to make a meaningful statement about, say C (massive code bases) vs. Modula-3 (very small code base) based on reliability, error density, etc.

Another angle, and one which I strongly prefer, is to boil it down to some simple and hard questions, somewhat similar to a security reduction (“RSA’s security can be seen as being defined by factorization hardness”).
One such question is whether a code text in a given language can be transformed into a form that can be processed by SAT/SMT. C can’t (it’s ambiguous), Modula-3 can (actually its library has been statically verified). Of course, Ocaml can, too.
Another question I like to ask is whether a language has both, proper(!) control structures (such as loops) – and – whether it allows logical (vs. value based) definition. The Wirth languages and derivatives do, C doesn’t.

Well noted, First/Last based arrays, loops, etc. alone can avoid a massive amount of bugs.

So, yes, It seems to me that I’m quite objective in this question (which also shows in me saying good things about Ocaml although I don’t like it).

As for your script question: Of course, I do not do the whole she bang for each and every line I write. But that isn’t arbitrary; it depends on the complexity of a task, on one-off vs repeatedly used tool, on for-me vs. for client, etc.
And luckily it’s less of a black/white decision than one might think; there are option in the middle, such, e.g. strongly typed scripting languages.

John Glat • April 14, 2017 12:32 PM

@Schneier

[[[ Aside from Russia and China, I can’t think of any other countries that would fit the bill. And in 2016, Russia more than China. ]]]

My firewall logs tell me China. Not Russia.

Patel Nisha • April 15, 2017 6:08 AM

In many ways Microsoft doesn’t get enough credit for long term support. Maybe Microsoft is trying to wane that from Windows users by pushing updates and dropping support early. Win 10 1511 will already lose support in May. But clearly they have been troopers in supporting XP, Vista, Win 7.
Apple is a mish mash of support, sometimes I am impressed, other times not so much. Google is a disaster when it comes to support. Android, Chrome OS both don’t support a product long enough. Android is fragmented badly and Chrome devices can loose support after 5 years from device release.

Moose • April 16, 2017 10:44 AM

There’s been another iteration in the Shadow Brokers’ releases:

https://www.lawfareblog.com/shadow-brokers-redux-dump-nsa-tools-gets-even-worse

“the really amazing stuff: operational notes from the NSA’s active targeting of banks in the Middle East and the NSA’s collection of Microsoft Windows exploitation tools. This may well be the most damaging dump against the NSA to date, and it is without question the most damaging post-Snowden release.”

Although there is some good news. From the end of the article:

“UPDATE II: Strike that previous update. Further reporting suggests that the mysteriously delayed Microsoft Patch Tuesday actually included patches for these tools, which suggests that the NSA did notify Microsoft in January when the Shadow Brokers released a directory listing suggesting they had these tools.”

You can pick up a copy of the latest dump here:
https://github.com/x0rz/EQGRP_Lost_in_Translation

If you’re running XP or Vista, you may have a problem:
https://www.theverge.com/2017/4/15/15311846/microsoft-windows-shadow-brokers-nsa-hacks-patched

Johnny Boy • April 17, 2017 11:07 PM

@ Mr. Schneier said,

“Mostly it’s Occam’s Razor. I am assuming that this was a sophisticated operation, and 1) the work of a government with a sophisticated cyber-intelligence operation. I am also assuming that it is 2) a country whose intelligence agency is working against the United States and 3) would be willing to embarrass the United States. (That third part ruled out Israel.) Aside from Russia and China, I can’t think of any other countries that would fit the bill. And in 2016, Russia more than China.”

Lol, in your own post you proved yourself wrong twice, so using what ever “Razor” it is called, let’s hope you’re right this time.

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.