Schneier on Security
A blog covering security and security technology.
« Camouflaging a WWII Factory |
| Friday Squid Blogging: Squid Robot »
October 16, 2009
The Commercial Speech Arms Race
A few years ago, a company began to sell a liquid with identification codes suspended in it. The idea was that you would paint it on your stuff as proof of ownership. I commented that I would paint it on someone else's stuff, then call the police.
I was reminded of this recently when a group of Israeli scientists demonstrated that it's possible to fabricate DNA evidence. So now, instead of leaving your own DNA at a crime scene, you can leave fabricated DNA. And it isn't even necessary to fabricate. In Charlie Stross's novel Halting State, the bad guys foul a crime scene by blowing around the contents of a vacuum cleaner bag, containing the DNA of dozens, if not hundreds, of people.
This kind of thing has been going on for ever. It's an arms race, and when technology changes, the balance between attacker and defender changes. But when automated systems do the detecting, the results are different. Face recognition software can be fooled by cosmetic surgery, or sometimes even just a photograph. And when fooling them becomes harder, the bad guys fool them on a different level. Computer-based detection gives the defender economies of scale, but the attacker can use those same economies of scale to defeat the detection system.
Google, for example, has anti-fraud systems that detect and shut down advertisers who try to inflate their revenue by repeatedly clicking on their own AdSense ads. So people built bots to repeatedly click on the AdSense ads of their competitors, trying to convince Google to kick them out of the system.
Similarly, when Google started penalizing a site's search engine rankings for having "bad neighbors" -- backlinks from link farms, adult or gambling sites, or blog spam -- people engaged in sabotage: they built link farms and left blog comment spam linking to their competitors' sites.
The same sort of thing is happening on Yahoo Answers. Initially, companies would leave answers pushing their products, but Yahoo started policing this. So people have written bots to report abuse on all their competitors. There are Facebook bots doing the same sort of thing.
Last month, Google introduced Sidewiki, a browser feature that lets you read and post comments on virtually any webpage. People and industries are already worried about the effects unrestrained commentary might have on their businesses, and how they might control the comments. I'm sure Google has sophisticated systems ready to detect commercial interests that try to take advantage of the system, but are they ready to deal with commercial interests that try to frame their competitors? And do we want to give one company the power to decide which comments should rise to the top and which get deleted?
Whenever you build a security system that relies on detection and identification, you invite the bad guys to subvert the system so it detects and identifies someone else. Sometimes this is hard -- leaving someone else's fingerprints on a crime scene is hard, as is using a mask of someone else's face to fool a guard watching a security camera -- and sometimes it's easy. But when automated systems are involved, it's often very easy. It's not just hardened criminals that try to frame each other, it's mainstream commercial interests.
With systems that police internet comments and links, there's money involved in commercial messages -- so you can be sure some will take advantage of it. This is the arms race. Build a detection system, and the bad guys try to frame someone else. Build a detection system to detect framing, and the bad guys try to frame someone else framing someone else. Build a detection system to detect framing of framing, and well, there's no end, really. Commercial speech is on the internet to stay; we can only hope that they don't pollute the social systems we use so badly that they're no longer useful.
This essay originally appeared in The Guardian.
Posted on October 16, 2009 at 8:56 AM
• 29 Comments
To receive these entries once a month by e-mail, sign up for the Crypto-Gram Newsletter.
The vacuum cleaner full of dna evidence idea was also done in Charles Stross's book Accelerando. In this case the perpetrators (who happened to be mafia enforcers for a RIAA type organization) vacuumed up dust and crumbs from a city bus then sprayed it all over the crime scene.
Strikes me as a certain amount of overkill, unless your blood somehow ends up covering the crime scene then it's highly unlikely DNA evidence will be bothered with. Real life isn't like CSI.
The solution to an arms race is well known.
It stops when the cost of the next stage sufficiently out weighs any potential benifit.
This was seen with ECM / ECCM / ECCCM systems.
In every case an arms race is an "evolutionary dead end".
The success stories are based on agility, that is run with a nearly new idea and drop it as the S-Curve tops out and jump on the next rising S-Curve as it starts to rise.
By S-Curve hopping you stay ahead of the evolutionary dead end that all arms races eventualy turn into.
This kind of sabotage is the reason why I do NOT believe in the much touted "semantic web" :
If anyone has anything to gain by polluting the semantic information, it will occur. Even in relativelly "good faith" environment such as research / science papers I believe there will be people who will spam the system to increase their professional visibility. Now imagine this in a commercial environment, failure is garanteed short of a policing Turing level AI.
So bad guys evolve in response to greater challenges? And this surprises people?
@ john greco,
"Strikes me as a certain amount of overkill... ...it's highly unlikely DNA evidence will be bothered with. Real life isn't like CSI."
It is when the "price is right".
If you steal 50million USD or murder a highly visable personality (Politico or Celeb) or commit an act for which there is a societal outrage, then there is a "political imperative" to solve the crime quickly.
Often this has led to wrongfull convictions but at the time "Justice is Seen to be Done".
You could take a leaf out of the books and Bruce's artical if you are planning such a crime...
Deliberatly leave some areas of your chosen crime scene alone so you do not leave your DNA or other trace evidence there.
Then dump your "vacuum cleaner full of dna evidence" over some of your touched crime scene areas as well as the areas you have deliberatly left untouched.
Then spray your touched crime scene areas with ordinary house hold bleach or other similar easily obtained cleaning agent (sodium hyperchlorate, sulfuric acid etc) as well as a small overlap into the untouched areas.
The result is a crime scene where it looks like the criminals tried to cover up their tracks but accidently missed bits...
As Bruce noted you can get quite a bit of milage from these ideas, but not just on the "Commercial Internet" ;)
"With systems that police internet comments and links, there's money involved in commercial messages so you can be sure some will take advantage of it."
This, it seems to me, says that Google/Yahoo etc. are attacking the wrong part of the problem. They should be attacking the financial incentives, rather than chasing after every tactical innovation thought up by every fraudster. Which is to say, instead of spending so much effort detecting fraud, they should be designing the system so that the incentive to commit fraud goes away.
I don't have a specific proposal (surprise), but making revenue proportional to clicks is certainly not the right way to do this.
There's also (sometimes) a predator/prey cycle that runs its course, although perhaps too slowly to be of much use. As a particular medium gets saturated with spam and spam-like things, people stop using it, and so it becomes less useful to spammers (albeit still useful to those who prey on ignorant would-be spammers). At some point the resources devoted to making the medium unusable decline to the point where fairly simple mechanisms can cut through the remaining spam, and some intrepid souls can begin having conversations again. Some parts of usenet, for example, are nearly habitable.
Maybe we should call this Berra's Law...
I just found this cute gadget in the 2009 Dow Jones event. The thing the intrigued me is called "UmiKey", it allows each user to stay anonymous and still being identified as an individual.
The most impressive is its portability on any PC, even Linux, convenience, low cost and it is truly plug-in to log in.
I think there are two other significant points:
1. The authorities (however defined) must be policing the domain (however defined) and subduing unauthorized actions. This may seem self-evident, but if there isn't any policing (i.e. enforcement of policy) by an authority, then there either won't be an incentive for such fraud, or the non-fraudulent participants will evolve their own system. There may still be incentives for other kinds of fraud, just not the "frame your competitor" type.
2. The perpetrators must have some expectation of anonymity. That is, there is some reasoning that convinces them they won't be identified or caught, or if caught, the sanction will be less than the profit. As a mental exercise, if every action taken could be unambiguously identified with a responsible agent, and the minimum penalty for infractions was a year in prison, you'd have a very different system. I'm not suggesting it'd be better, just very different.
@LOL at October 16, 2009 1:02 PM
So what? I have a masters in Mangaement Information Systems and bachelors in Ecomonics and Business Administration and Computer Science and Information Systems, i'm a Certified Internal Auditor, a Certified Information Systems Auditor, and very soon I'll be Certfied Financial Services Auditor and a Certified Fraud Examiner.
Not a single one of those degrees or the study for a profesional designation taught me nearly as much as my years in the trenches. And Bruce has been in more trenches than I.
Degrees and designations may prove someone is knowledgable, but knowledge means little if you can't apply it.
@dot tilde dot
Interesting. DNA contaminated cotton swabs made it into CSI:NY this week.
Good article Bruce.
Had thought about how contaminated DNA evidence could be misused in a crime scene, and actually read something similar in a (pretty awful) book years ago - Iain Banks comes to mind. I'm guessing that there are ways to determine what is contaminant DNA and what is the DNA of a suspect?
The idea of using adsense's antifraud mechanisms or a spam filter against a competitor however, I had never thought of - and has definitely got me thinking in different ways.
But vast majority of criminals are really really dumb. Evil masterminds are very rare - people smart enough to do smart crimes, are smart enough to earn loads of money in other less risky ways.
So any technology that makes it easier to identify and catch criminals might result in a major reduction in crime.
They're theoretically possible to abuse, but so far a lot more innocent people were freed thanks to DNA evidence who would have otherwise been convicted, that number of innocent people who got convicted due to DNA evidence forgery.
I feel pessimistic about this one.
While your statement on the number of "innocent people who got convicted due to DNA evidence forgery" sounds nice, we don't really know, do we? Unless the perpetrator confesses, the forgery stands because people believe it must be true.
Our criminal justice system has built itself on on this continual search for the absolute truth - being able to point to something as "incontrovertible"; whether it be eyewitness identification (worked really well in the Salem Witch Trials, etc), fingerprint evidence (well, subject to the interpretation of an analyst, of course), and now DNA evidence.
Ten years from now, when the full knowledge of what we've done in the last fifteen or so years becomes evident by the development of more-refined techniques, technologies, and knowledge of the subject - then we can judge your statement as fact or wishful thinking.
@ B. Real,
"Our criminal justice system has built itself on on this continual search for the absolute truth - being able to point to something as "incontrovertible"; "
Err no, it has continualy sought to reduce to "words on paper" the physical world and what happens in it.
This is so that a judge can take the pieces of paper and decide if they follow the rules or not. If they do then a jury is then alowed to consider the content of those pieces of paper as presented by a witness.
The witness is cross examined by people who have little or no knowledge as to the actual truth or falseness of the words on the paper.
Thus the jury sees a side show and rates what is presented against their own prejudices for those witnessess and the "performing arts" abilities of those cross examining...
This apparent "searching after the truth" is a nonsense not just from the legal paper chase but also from the physical limits of the universe as we currently understand it.
"Ten years from now, when the full knowledge of what we've done in the last fifteen or so years becomes evident by the development of more-refined techniques, technologies, and knowledge of the subject"
The reality is that in most cases the science behind the investigation of crime cannot realy go any further. In many cases the tests available are more sensitive than the background noise level of that which they aim to measure.
That is they have reached the point of giving us "the square root of bugger all" in determining if something is there let alone how it got there.
The areas where the science is improving is in the "showmanship" side.
That is people are looking for ways to stop cross examination argument being made to sell product.
For instance as a "scientist" you are aware that various arguments can be made but they are in the main pointless to other scientists.
But they are great for legal types to argue about in cross examination.
You therefore find a solution for a non problem that is not realy there take out a patent and build up the scary side of the non problem to sell your patented product...
Oh and a great way to do this is supply it as a plot line to a "CSI" type plot writer...
The simple fact is that all the tests in use are "gamable" in one way or another and often the attempt to close one gamable aspect opens up a fresh way to game the system. And so the game goes on.
But at the end of the day all these tests do is show something is present and possibly in what quantity. They do not and cannot explain how they got there or why.
Thus "this continual search for the absolute truth" and supposadly "being able to point to something as incontrovertible" is a "fools erand" in the same way as "searching for the end of a rainbow to find the crock of gold".
Many people make the claim that an arms race is a waste of time and just a dead end. There's plenty of evidence to the contrary though, particularly in coevolution. When Danny Hillis was doing his evolutionary algorithms on massively parallel computers, he noticed that the best solutions were found when he introduced competitive "parasites" into the gene pool. And the existing solutions were found faster. In the words of one report, the arms race "turbocharged" evolution, rather than retarding it. (link below)
This isn't just for evolution though. It's competition in general. Top Coders is another example. They get the requirements from business, and let several of their many programmers have at it. Whoever produces the best solution gets paid, and their code is added to the repository. Many studies and real-world companies like Top Coder have shown that competition, like diversity, improves quality. There are ways it can go wrong, but if done in a certain way it often trumps mere cooperation.
So, I think arms races are far from a waste of time in and of themselves. Arms races can be the best way to improve a product. The important thing is that they are racing towards the right goal. The arms race between antivirus and virus authors is an example of a wasteful arms race: antivirus side would be better off investing in proactive measures like SDL methodology or more secure OS's. But the arms races and competitive methodologies pay off in other schemes, including those NASA projects you keep citing.
Hillis's arm's race: http://www.generation5.org/content/2000/...
@ Nick P.,
"Many people make the claim that an arms race is a waste of time and just a dead end. There's plenty of evidence to the contrary though, particularly in coevolution."
Ahh the problem I think is one mans proto "arms race" is another mans "out evolving".
You could look at it this way,
A preditor A attacks a beast B.
The beast has several options amongst which are develop a thicker hide or move faster than other beasts.
Initialy either is a reasonable stratagy. However either can also become an arms race, it depends on what the preditor A does.
It A goes on to attack other beasts then neither stratagy has become an arms race.
If however A develops longer teeth for B's thicker skin then B might just develop thicker skin again in which case A develops even longer teeth, it has become an arms race.
Likewise if A becomes more fleet of foot to B's increase in speed then B might just carry on getting faster and likewise A might just get faster in turn, again it's another arms race.
Out evolving is B using a tactic different to a previous tactic such that A is nolonger a threat to B.
Diversity of response is benificial in a competative arena it increases development of both A and B. Non diversity in response from both A and B is an arms race and detrimental to both and is an evolutionary dead end.
An example of this where dinosaurs that developed thicker and thicker surface bone structures to deter large preditors. The large preditors developed larger and more powerfull jaws and teeth to compensate. The cycle continued without change and was thus an arms race between the two.
However they both became slow and cumbersom. The preditor gave up speed and agility for brut strength and large teeth and thus could only feed off a diminishing subset of prey. And the heavily armourd dinasour became easy pray to small agile dinosaurs that learnt to hunt in packs and could get at the soft underbelly easily.
In essence an arms race is over specialisation giving rise to either diminishing returns or fragility to changes in the environment.
The arms race is only half the story -- the other half is the distinction between "active" and "passive" defenses. The former can react and adapt to attacks; the latter are cheaper, but much more vulnerable to persistent attack.
The key thing is, in modern times, any automatic (and weakly supervised) system is effectively a passive defense! This is a problem for people who are facing cost constraints....
@ Nick P.,
"Hillis's arm's race"
On reading it two things stuck out,
One was that random genetic type algorithms rarely if ever reach a minima within a bounded time constraint (which nature usualy does every time).
Secondly that Hillis's arm's race is actually designed to reach a single optimal solution within a given set of constraints.
This second point is interesting simply because it is the opposit of noramal stratagy of survival, which is to be as flexable as possible, thus lacking specialisation dead ends.
Hillis's was looking specificialy for one optimal and efficient algorithum for a single given purpose (searching).
Biologicaly this is the equivalent of the ATP solution to providing energy in all living cells. However as a solution it could not be reached by simple chemistry alone thus has been a bit of a puzzle to the evolutionary process (it appears that the work of Peter Mitchell on what he called chemiosmosis that led to his 78 nobel prize is probably correct).
More interestingly "Hillis's arm's race" idea has been refound by those doing chemical crystalography.
Back in the mid 90's Mike Payne of Cambridge University had a program called CASTEP which simulated what atoms did in solids such as crystals.
However it used to hit the non optimal minima problem and it's results where not the regular crystal structures that nature produces.
However some ten years later Chris Pickard of my old stamping ground UCL whilst helping one of his students reworked CASTEP and ran it on considerably more powerfull hardware and found that he was starting to get sensible output not the previous non optimal mess. The reason apears in part to being able to reconfigure the starting positions of the atoms.
However others have been working on similar ideas using anealing algorithms where the temprature gradient is constantly raised and lowered, which oddly appears to be the best way to physicaly synthasize DNA.
And more recently genetic algorithums have been used to take many non optimal minima and "cross breed" them killing high minima offspring and keeping those at or below the minima of the parents. This to has led to regular crystal formation.
However we still do not know how nature gets the optimal minima each and every time for crystals.
When we do I think we will have a bit of a shock for the mathmatical community and such questions as N=NP etc will "drop out naturaly".
And likewise of course it will also have significant implications for the factoring of numbers as finding the two primes is an optimal low state problem...
Maybe we should take a leaf out of Leonard M. Adleman's book and instead of using his 94 DNA computer (random) we should jump over anealing systems and should look at growing crystals. It might be more fruitfull than quantum computers.
5 or so years ago in Bulgaria, robbers drilled a hole in the floor, robbed a bank office, and filled all rooms, including their entry point with hair collected from a number of hair saloons. At the end, they damaged the sewer pipes and flooded with feces. Beat that! :-)
I read Halting State a couple of weeks ago after spotting it on the shelf at a mainstream book shop. Glad to see that it got a mention here! I'd recommend it to anyone who enjoy's William Gibson's work. It reads a bit like his newer present day books but with some of the technological bent of his older work.
@ Nikolay Kolev,
"At the end, they damaged the sewer pipes and flooded with feces. Beat that!"
Such a task for such a short time, it must have been a Herculian effort (in reverse).
In natural evolution, fitness is whatever survives. In directed evolution, fitness is whatever the experimenter says it is.
But, as others have alluded to, incredible effectiveness at the detection of spam, joe jobs, metaspam metaframes usw is not something anyone needs absent the idiots trying to get our attention. Perhaps if the algorithms developed to do this kind of disambiguation had applications elsewhere there might be an overall fitness gain, but instead what we've seen is mostly adaptations of disambiguation methods already known from toher, more useful fields.
@Clive: "It stops when the cost of the next stage sufficiently out weighs any potential benifit."
In the predominant arms race of my lifetime, the nuclear arms race between the US and the Soviet Union, this was clearly not true. There was clearly no benefit to either side from the production of additional weapons. (Once you can destroy the entire world once, how many more do you need?)
The nuclear arms race ended because one side was no longer able to continue. It did not end due to a choice based on a cost-benefit analysis.
The "arms race" stops whenever a business is smart enough to realize it's wasting money and time advertising on the internet.
I have responded to perhaps five internet ads since the internet existed. All five ads were on small, trustworthy web sites that carried only a few ads appropriate for the audience. All other ads are ignored or (even better) blocked. Any site that puts up too many hard-to-block ads is never visited again. Perhaps I am atypical, put almost no advertiser wins when I'm surfing.
"The nuclear arms race ended because one side was no longer able to continue."
Yup that appears to be in full agrement with my comment,
"It stops when the cost of the next stage sufficiently out weighs any potential benifit."
"It did not end due to a choice based on a cost-benefit analysis."
I'd beg to differ, it stopped because it became abundantly clear to all that "perceived security at any price" was bankrupting the country and it had got to the point where the inevatable had been obvious to the average drunk on a Moscow street for many years.
When anybody questions the desirability of an on going process they are doing a "cost-benefit analysis" (ie what's in it for me). It might not be a "formal analysis" of X Rubles for Y nukes / tanks / etc analysis but as simple as "I can't get a winter coat because the Military take all the cloth for uniforms".
The people at the bottom are usually the first to "cost-benefit analysis" because they are the first to feel the pain of shortages and the pain is a cost that focuses the mind.
Schneier.com is a personal website. Opinions expressed are not necessarily those of BT.