This talk by Dan Geer explains the NSA mindset of “collect everything”:
I previously worked for a data protection company. Our product was, and I believe still is, the most thorough on the market. By “thorough” I mean the dictionary definition, “careful about doing something in an accurate and exact way.” To this end, installing our product instrumented every system call on the target machine. Data did not and could not move in any sense of the word “move” without detection. Every data operation was caught and monitored. It was total surveillance data protection. Its customers were companies that don’t accept half-measures. What made this product stick out was that very thoroughness, but here is the point: Unless you fully instrument your data handling, it is not possible for you to say what did not happen. With total surveillance, and total surveillance alone, it is possible to treat the absence of evidence as the evidence of absence. Only when you know everything that *did* happen with your data can you say what did *not* happen with your data.
The alternative to total surveillance of data handling is to answer more narrow questions, questions like “Can the user steal data with a USB stick?” or “Does this outbound e-mail have a Social Security Number in it?” Answering direct questions is exactly what a defensive mindset says you must do, and that is “never make the same mistake twice.” In other words, if someone has lost data because of misuse of some facility on the computer, then you either disable that facility or you wrap it in some kind of perimeter. Lather, rinse, and repeat. This extends all the way to such trivial matters as timer-based screen locking.
The difficulty with the defensive mindset is that it leaves in place the fundamental strategic asymmetry of cybersecurity, namely that while the workfactor for the offender is the price of finding a new method of attack, the workfactor for the defender is the cumulative cost of forever defending against all attack methods yet discovered. Over time, the curve for the cost of finding a new attack and the curve for the cost of defending against all attacks to date cross. Once those curves cross, the offender never has to worry about being out of the money. I believe that that crossing occurred some time ago.
The total surveillance strategy is, to my mind, an offensive strategy used for defensive purposes. It says “I don’t know what the opposition is going to try, so everything is forbidden unless we know it is good.” In that sense, it is like whitelisting applications. Taking either the application whitelisting or the total data surveillance approach is saying “That which is not permitted is forbidden.”
We all know the truism, that knowledge is power. We all know that there is a subtle yet important distinction between information and knowledge. We all know that a negative declaration like “X did not happen” can only proven true if you have the enumeration of *everything* that did happen and can show that X is not in it. We all know that when a President says “Never again” he is asking for the kind of outcome for which proving a negative, lots of negatives, is categorically essential. Proving a negative requires omniscience. Omniscience requires god-like powers.
The whole essay is well worth reading.