Schneier on Security

Kevin • January 25, 2007 9:27 AM

It was not until I’d had a graduate course in datamining that I truly appreciated Bruce’s criticisms thereof. Datamining can only yield useful statistics, and can only benefit on a statistical level. It might, for example, help a retailer decide to cluster various items in a way that boosts revenue in candybar sales by 5%. It will not allow the retailer to prevent Alice from purchasing the cheap dog food and nothing more; it will not ensure Bob will grab a 6-pack of beer with his frozen pizza. It might increase early detection of various cancers via genetic screening, reducing the overall cancer fatality rate by 10%; it will not say, at birth, which of Carl, Dave, or Earl will develop cancer (although based on a vast collection of data including true positives and true negatives; a cost of screening, a cost of false positivies; a cost of false negatives; datamining might say that after the age of 30, Carl should be screened every 3 years; Dave every 10 years; and Early every 20 years).

The NSA does not have nearly the data needed to even make that kind of prediction. Plus, I’m guessing the NSA isn’t looking to datamining for statistical reductions in terror-related fatalities; they want it to sift through data of 300 million Americans and say, “These 6 guys are plotting to blow up the Sears tower!”. I’m also guessing they will pin all of their inevitable failures on, you guessed it, insufficient data due to cumbersome judicial oversight processes.