Schneier on Security

Roger • September 11, 2006 1:56 AM

After playing for a while, I was getting annoyed at endless repetition of some of the boring pictures, and too-infrequent occurrence of the best ones, so I wrote a bot to snarf the whole list so I could browse them sequentially (in order to be a good neighbour the bot just snarfs the image names, and doesn't download the images at all).

But from its results some curious features turned up. Probably they signify nothing but a curious defect in the PRNG, but other readers might be interested [1]:
1. rate at which individual pictures are served up is distinctly non-uniform. Some pictures are served up an order of magnitude more often than others [2]. I get the impression that some are also more likely to follow certain others, but my bot data doesn't allow me to quantify this.
2. roughly 37% of pictures are tagged as being bombs, but pictures of bombs are only served up 1 time in 6;
3. I had thought that images served up more often would be identified correctly more often, due to players learning them. This doesn't appear to be the case; there doesn't seem to be any particular relationship. This may suggest that most players play for a period too short to see most images (about 140 at the moment, but growing all the time);
4. non-bomb images are more likely to be identified correctly than bomb images (80% vs. 65%);
5. the most-often correct bomb and non-bomb images get about the same score (92 and 94%) but the least-often are widely separated (22 and 35%).
6. However, both the gaps in both averages and minima are getting narrower every time I run my bot (the first time I ran this, this least-often correct bomb image was correctly identified only 7% of the time).

Points 4 and 5 tend to suggest that most people are better at corectly identifying common, harmless objects than at seeing the hidden hazard in novel objects (this isn't really surprising).

Point 6 could be explained if most players play a lot, and gradually memorise the results (i.e., get trained, just as is hoped for in the process being spoofed). However point 3 tends to suggest that this may not be the case; people don't seem to do better with images they have seen before than with ones they haven't. In that case, other explanations include:
a. Badly written bots which vote maliciously or randomly [3];
b. Players who vote randomly.
Random voting will tend to cause all the results to gradually converge over time. A malicious voter might attempt to deliberately distort votes for some political reason.

_____
1. I culled the stats for a couple of controversial images; not just the goatse one (which still hasn't been removed) but also those for which there are complaints of incorrect categorisation.
2. The most common bomb image is the easily-identified atomic bomb. The most common non-bomb image is the highly deceptive "Hills bomb", which is a grainy photograph of a model of a bomb.
3. I confess the first cut of mine did this, until I realised it might distort stats. Now when it encounters an image it has seen before, it votes correctly in proportion to the number of correct votes reported last time it saw it.

Roger • September 13, 2006 5:00 PM

The Bomb-or-Not site has now been down for two days.