Lessons in Trust from Web Hoaxes

Interesting discussion of trust in this article on web hoaxes.

Kelly's students, like all good con artists, built their stories out of small, compelling details to give them a veneer of veracity. Ultimately, though, they aimed to succeed less by assembling convincing stories than by exploiting the trust of their marks, inducing them to lower their guard. Most of us assess arguments, at least initially, by assessing those who make them. Kelly's students built blogs with strong first-person voices, and hit back hard at skeptics. Those inclined to doubt the stories were forced to doubt their authors. They inserted articles into Wikipedia, trading on the credibility of that site. And they aimed at very specific communities: the "beer lovers of Baltimore" and Reddit.

That was where things went awry. If the beer lovers of Baltimore form a cohesive community, the class failed to reach it. And although most communities treat their members with gentle regard, Reddit prides itself on winnowing the wheat from the chaff. It relies on the collective judgment of its members, who click on arrows next to contributions, elevating insightful or interesting content, and demoting less worthy contributions. Even Mills says he was impressed by the way in which redditors "marshaled their collective bits of expert knowledge to arrive at a conclusion that was largely correct." It's tough to con Reddit.

[...]

If there's a simple lesson in all of this, it's that hoaxes tend to thrive in communities which exhibit high levels of trust. But on the Internet, where identities are malleable and uncertain, we all might be well advised to err on the side of skepticism.

Posted on May 23, 2012 at 12:32 PM • 13 Comments

Comments

paulMay 23, 2012 12:54 PM

In communities that pride themselves on toughness and skepticism, it's also important to plan hoaxes that are in line with the community's conventional wisdom and "intuition".

Barry KellyMay 23, 2012 12:57 PM

"Erring on the side of skepticism" is too drastic, IMO. Rather, we should increase social penalties for people who perpetrate hoaxes. Ban them; shame them; create registries to track them as they move from one group to another.

Our lives are diminished when we decrease our trust. Our experiences get mediated through filters of cynicism, the real becomes more distant, and we're less inclined to change and learn.

I think it would be wrong to apply too many of the lessons of computer security to social "security". Given a choice between a network of people who trust one another, risk being fooled, but apply harsh penalties to those caught doing the fooling; and a network of people who do not trust one another, and apply tests and filters to all actions: I would choose the former every time. The latter is a sick society, poisoned by its mistrust, and should be abandoned and left to die off.

Think of a village analogy. Is it better to live in a community where you need to make sure all your doors and windows are locked at night, lest you be robbed? Or one where you can sleep easy, even with an open back door? If you live in one of the former, you should try living in one of the latter some time. It's less stressful all round.

Brandioch ConnerMay 23, 2012 12:58 PM

I saw that article previously and I was not very impressed.

It does not take much to build a Wikipedia entry on an obscure subject. Even a badly done one. Or to link it with other badly done entries on obscure subjects.

Who is going to verify the entries? Who cares enough to verify them? Who will even see them in the normal course of events?

"If there's a simple lesson in all of this, it's that hoaxes tend to thrive in communities which exhibit high levels of trust."

I don't get that. Anyone who trusts Wikipedia does not understand Wikipedia.

DanielMay 23, 2012 1:04 PM

I'm reminded of the cliche, "Everything you read in the newspaper is true except for the 10% you know from personal expertise to be false."

I don't know why this would be any less true for the internet.

Barry KellyMay 23, 2012 1:07 PM

Brandioch: Anyone who clicks on a Wikipedia link in a search result (i.e. approximately all WP visitors) trusts Wikipedia, for some value of "trust". And they should do so: because trusting (at, say, a 90% level) the content is more efficient overall than not trusting it (most citations on substantive cannot be easily verified because they reference hard copy books, for example).

fredMay 23, 2012 1:18 PM

Got a call from the ammyy_dot_com scammers recently. Their schtick (all delivered in a thick Hindi accent) is to say that they're from Verizon and need to fix your dns and would you please visit their website and do what they say.

I got a call from them once before and complained to their hosting service, but received a letter back from the lawyers saying they couldn't do anything, and ammyy is obviously still around.

So I pulled out the iPad, rerouted all traffic through some open proxy operating out of Tehran, identified my browser as Internet Explorer 6, and went ahead with the instructions that the urgent Indian provided. Sadly, their scam is to bruteforce a .EXE into the browser from the start, so I wasn't able to provide them with the identifying code they wanted and ultimately failed to send them on a goose chase through teh Iranian internets.

Does anyone know a better way to get at these clowns?

Brandioch ConnerMay 23, 2012 1:43 PM

@Barry Kelly
And they should do so: because trusting (at, say, a 90% level) the content is more efficient overall than not trusting it (most citations on substantive cannot be easily verified because they reference hard copy books, for example).

I don't think "efficient" is a useful standard here. Being efficiently wrong is still being efficient. And wrong. Particularly with regard to "hoaxes" like the one referenced.

Fred PMay 23, 2012 2:47 PM

@Barry Kelly - when I care about a Wikipedia entry being accurate, I at least look at some of the online references; if there are none that are useful, I use a different source. I also often look at the talk and occasionally the view history pages.

Dirk PraetMay 23, 2012 7:22 PM

If there's a simple lesson in all of this, it's that hoaxes tend to thrive in communities which exhibit high levels of trust.

I believe we need to differentiate between trust and gullibility. Although anyone - individuals and groups alike - can and will get fooled plenty of times, there are basic rules to verify and/or cross-check information obtained from any source, trusted or not. Hoaxes IMHO will thrive better when aimed at badly informed audiences or when they fit very well into the belief system of the targeted group, irrespective of it being self-adopted (conspiracy theorists) or imposed upon them (fear of terrorists).

A couple of years ago, some thinktank over here by way of an experiment managed to slide some really bizarre stories into mainstream news media without anyone checking the sources or reserarch behind them. One of those was a study alledgedly showing that right wing voters had a significantly better sexlife than their socialist and green party counterparts. Obviously, several conservative politicians were dumb enough to quote it in interviews and TV shows until the group behind the story revealed that it was completely bogus. It was a serious wake up call for both media and the general public.

@ Barry Kelly

Is it better to live in a community where you need to make sure all your doors and windows are locked at night, lest you be robbed? Or one where you can sleep easy, even with an open back door?

Try Saudi Arabia. Several friends of mine who worked there on contracts have told me that it is indeed uncommon for people to close doors and windows. This of course has little to do with trust, but everything with the fact that the local penal code deals rather harshly with thieves.

SpellucciMay 23, 2012 10:24 PM

@fred Does anyone know a better way to get at these clowns?

You did just the right thing: waste their time. The more we all waste their time the fewer people they can scam. Speak slowly, cooperate slowly, be hard of hearing, and generally keep them on the phone until you get too bored to continue.

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Photo of Bruce Schneier by Per Ervland.

Schneier on Security is a personal website. Opinions expressed are not necessarily those of Resilient Systems, Inc.