Alex Stamos on Content Moderation and Security

Really interesting talk by former Facebook CISO Alex Stamos about the problems inherent in content moderation by social media platforms. Well worth watching.

Posted on May 29, 2019 at 6:03 AM • 34 Comments

Comments

1&1~=UmmMay 29, 2019 12:05 PM

@Folks:

I know BoJo is a joke/waste, and all things considered does deserve a kick in "his seat" preferably into the neatherhells but this aint the best place, can we take it over to the latest squid, where we can have a proper "turkey roast" ;-)

IrritatedMay 29, 2019 1:52 PM

I thought squid stories are supposed to be "security stories that i haven't covered", according to Bruce, not "anything goes".

Jim WMay 29, 2019 2:14 PM

@Irritated Perhaps so but they sure have been some interesting conversations.

Petre PeterMay 29, 2019 4:29 PM

Cybersecurity education is difficult because, beside input validation, it tries to teach something you shouldn't do as something you should know how to do. Tradeoffs are problematic the user cannot make a choice - data isn't portable-the export button doesn't exist. This is fewdalism.

BobMay 29, 2019 4:57 PM

@Irritated

Is hijacking Western democracy not a security concern?

Anyway, on topic, I'd assume that most readers here would be aware of the issues with content moderation on sites that deal with massive quantities of user-generated content. Seems like politicians and soccer moms are really the only ones who think that if we just nerd harder, we'll come up with a magical technical solution.

IsmarMay 29, 2019 6:14 PM

One part of me is saying:
As they say - nothing is free, and if it cannot be moderated efficiently then maybe that type of platform should not exist?
The other part is , however, saying so what that it cannot be moderated, let them express themselves anyway they like and let us make a decision as what to make of it.
This blog, IMHO, has the right balance between the two which is not kept like so by the moderator as it is enforced by the blog’s community (which still lets me do my ranting from time to time 😀)
The only thing about larger platforms is that it seems that larger the community the more difficult the community moderation becomes

Jesse ThompsonMay 29, 2019 7:46 PM

Well, I listened through the whole presentation so I can offer some direct insights.

Alex seems like a sharp guy with a pretty broad perspective, which is refreshing. But if you'd like to see some of his biases, you can check out 40:59 where he freaks out about 4chan (or one of it's chan brethren, I can't be bothered to find out which one) failing to be part of the approved borg content control collective. Or 44:40 where he does a pretty good job describing what regulatory capture is, how it works and evolves, but appears to spin that as a good and desirable thing.

I'll say the same thing I said (twice) about our host's "public interest technologist" essay(s): there exists a really impressive danger in forgetting that there does not and cannot exist one "correct" sense of morality and public good that some select, elite group of situationally empowered people are somehow a reliable authority on.

Morality and public good are and have to be a contested quality, and every person has to have some influence over that conversation.

Trying to coral all discourse through an approved set of corporate filters, and then relying upon the byzantine expense of bureaucratic compliance to stifle all competition and further concentrate power kind of leads towards the very autocracy that Stamos claimed to be trying to protect against to begin with.

I think it's easy to view oneself as championing free expression simultaneous to misidentifying what qualifies expression as even being free in the first place. "Free to conform to my expectations" just doesn't cut it. Stamos tries to chastise PRC for doing that, but I feel that he leaks a bit of that perspective himself in the process.

Ross SniderMay 29, 2019 8:14 PM

In fairness I didn't have time to watch the video above. I don't have much time to comment this evening.

Moderation communities seem promising to me, but I usually find moderators are staffed with incentives and tools that don't quite make for the moderation best for the community. From what I can tell there's some integrations with intelligence and policy systems, as content running counter to current propaganda campaigns is taken down very quickly. Surprisingly OKCupid and other dating website volunteer content moderation seems to be well done and also not very controversial. It's the "opinion" websites that are the most contended because they are mixed into the information warfare front.

I've been politically censored on essentially every moderation community at some point in time: hacker news (for arguing China's points and against Hague arbitration of the South China Sea), reddit (for spreading information about US propaganda and support for terrorism in the Syrian conflict), Facebook (for spreading information about the Snowden documents), and news websites (various themes). Mostly things that get close to national security issues, or calling out US domestic propaganda efforts. I suspect that's not entirely from an overton window effect - there's probably a fair bit of integration with intelligence/military efforts.

HumdeeMay 29, 2019 8:26 PM

@jesse

I agree that morality is contestable but that does not mean there should be no rules to the contest. As i like to say: i believe in free thought and free speech but not the freedom to yell.

WokMay 29, 2019 8:34 PM

@Humdee

Can you be more specific to not have the "freedom to yell"? Because there's a lot of people with elevated soap boxes I'd like knocked down a few pegs but that's unlikely what you're thinking about.

MarkMay 29, 2019 10:40 PM

Probably the same problem as this site. Leftwing politics remain, anything to the right of Marx gets deleted.

BoJo needs to take the UK out of the EU ASAP. The EU in an anti-democratic institution. Democracy means democracy. The people have spoken. Brexit means brexit.

Dunning-nessMay 29, 2019 11:54 PM

"Leftwing politics remain, anything to the right of Marx gets deleted."

So true. It's not that you're unrealistic, it's the WORLD's problem of factual perspective!

Thanks for explaining why disinformation shows equal promise to neo-Conservatism, being related phenomenons.

No sarcasm at all, he believes this.

Clive RobinsonMay 30, 2019 2:09 AM

@ Ross Snider,

Moderation communities seem promising to me, but I usually find moderators are staffed with incentives and tools that don't quite make for the moderation best for the community.

Moderating communities is always going to be hard.

For instance see comments just today,

https://www.schneier.com/blog/archives/2019/05/alex_stamos_on.html#c6793380

https://www.schneier.com/blog/archives/2019/05/alex_stamos_on.html#c6793391

This thread is not about (party politics) or (idiot politicians) yet those two commenters who have not made a constructive comment to this thread (or in one case ever on this blog under that handle) think it's OK to derail the thread entirely.

Ironically they breach one set of rules designed to keep a thread on topic yet make complaint about the rules...

But they don't see or I guess want to see why that might be an issue for others... Which is an issue psychologists have a number of names for, because it's actually quite common.

Over the millennia as society has developed, rules and sanctions have developed for what some call the "meatspace" or tangible, physical world and how people behave in it. Technology has given us the Internet and opened up "infospace" considerably more rapidly than society can react thus rules and sanctions are few to none.

Our host @Bruce even after having nearly had his blog trashed by extremism and partisan political venom and rhetoric dressed up as faux patriotism, is still more tolerant of "party propaganda" type comment than many would be. For which many are grateful, myself included, and for which he should be commended.

Unfortunatly security especially ICT Security is becoming not just broader in scope, due to the many very public failings it's had, it's no longer a "nerd only domain". Mainly this is because of the very real effect it's having on the everyday citizen. After all ask yourself of the aproximately 330million Americans in the world, how many have not been effected directly or indirectly by the failings of ICT Security?

Thus the ICT Sec tent has become too interesting and very noticeable, thus the "Camel's nose" of "political interest" has forced it's self through the ICT Sec tent flap. And like "toppsie's cat" where ever it goes trouble is soon to follow. That is inevitable as part of the human condition, or if you prefer "just the way it is".

Thus the question arises of how you manage the intrusion that the Camel's Nose is? Beating it with a stick will only rile the beast up to the point it brings down further trouble if not the whole tent. The trick is to make the beast more intetested in other things, thus it will lose interest and hopefully find another tent more interesting. Unfortunately ICT Sec or more correctly those that employ it badly are still making lots of interest for the Camel.

Worse yet is that main stream politicians now see ICT as their way to "reach the people" in their usual totally inept way thus like children who despite being warned of the danger play with fire and not unexpectedly not just get burnt but immediately want to blaim others for their stupidity and petulance, and such behaviour is more catching than the common cold.

But others are more adept at using the technology in a more focused way. With any thing that is new that has few or no rules and sanctions, the worst of the worst form the exploiting vanguard, closely followed by other scheming opportunists. For some reason we are encoraged to think of politics as a simple line with a moderate center and extream ends. Unfortunatly it is a way way to simple model and has little contact with reality.

Thus you can have those who are allegedly politically moderate behaving in extream ways and happy to make use of not just the scheming opportunists but also the worst of the worst, who realy should have no place in polite society. In the past whilst such people existed their power was relatively small due to the lack of resources they had available to them and that they were frequently sanctioned. Thus their resource were limited and they could not access the same "force multipliers" others more acceptable to society could.

As I've pointed out a number of times in the past the tangible physical world can be viewed as a subset of the intangible information world. That is there are constraints in the tangible world that don't apply in the information world. I've also pointed out that being creatures of the tangible world most people make assumptions based on physical world interaction that are not true in the information world.

The example I use from time to time is that of the "distance costs" assumption subconsciously built into economic ideas that gives rise to the notion of competitive markets. When "distance costs", making your market coverage larger has significant costs which means you have to charge more for your tangible goods the further they are from the point of production. Thus somebody starting making similar products has in effect a competative advantage in their place of production (all other things being equall). Thus with "distance costs" several players can give rise to a market place where choice is available to consumers thus realistic competition is possible. The same is not true of information services where moving data has no distance cost to either the service provider or their customers thus you end up not with a market but a "first to market is the market winner". All others can do is create a different market for themselves, due to other effects these different markets get either absorbed or destroyed by those with older markets.

The result is you end up with just a handfull of very massive force multipliers with effectively zero cost of use open to anyone with out any effective rule or sanction...

Thus the worst of the worst and scheming opportunists now have little or no cost to enter the game. Thus we have seen the rise of the "first in market" effect promote the extreamists.

The owners of these force multiplying services do not want the trapings of the more traditional tangible goods markets, which due to history work within a lot of rules that they can not avoid. That is the force multiplier owners do not want the rules we more normally call regulation and legislation. Which at the end of the day are very much designed to ensure market players behave responsibly towards their customers.

Currently those who own these force multiplying services are trying to buy two forms of rules. Ones that mean they can make more money/profit/influence/power, and ones that keep others out of what they see as their domain in various ways.

One aspect of this behaviour is that they want to avoid being seen as publishers which incures all sorts of responsabilities but as "common carrier" communications providers which have effectively no responsability thus liability for what people use their infrastructure for. In reality the force multiplier service providers are with little doubt publishers by the very basic way they opperate[1].

The problem for the force multiplier owners is that as they "publish for everyone" at no direct charge, they can not aford to moderate the billions of articles they actually publish.

Whilst people have argued about the figures one thing that has come out of the congressional enquires into the Russian Factory output is that it is a fantastically small fraction of the traffic. Whilst not in the same league as the proverbial drop in the ocean, it is way to much information to be seen and evaluated conciously by humans, not just in real time but at all. Which is why it mostly does not get consciously seen (something the Advert industry knows only too well but won't admit to).

Thus those who own the force multipliers can not survive if they become regulated as publishers. Thus they have entered a devils pact with politicians. They will look only for and stop what the politicians don't want to have their attention drawn to...

This they can do by somewhat simplistic rules in software and by banning those they get complaints about. Thus it does not nor can not reflect what a moderate community would wish for.

[1] Common Service providers provide a "point to point" service not a "broadcast service". To get the "common service" status you traditionally had to promise "privacy of service" that is the old "Must not see, thus can not see, thus can have no responsibility for content" argument. A broadcaster is actually a publisher, they have no choice but to see what it is they are duplicating to multiple points of delivery, thus they have responsibility for what they broadcast no matter if it's stone tablets from on high through all writen and communicable forms even if it is from down in the gutter (and people wonder why I don't have a blog...).

OtterMay 30, 2019 4:24 AM

I stopped listening very early.

He says, the is no precident for deciding what people are allowed to do. He hasn't noticed, for example, the Bible, or even the most shallow history book. Or a schoolyard.

He says, nobody likes to make trade-offs. He appears to be talking about an ecology where the gate-keepers devote extraordinary expenses and efforts to hide the fact that there are important offs to trade.


Some people above say "cannot afford". This is not true. The truth is "choose not to afford". And "choose to afford", for example, crude censorship when more powerful gate-keepers choose to lean on them.

IrritatedMay 30, 2019 1:09 PM

@Clive Robinson

Not sure why you're targeting my comment in particular. It wasn't out of the blue - I was replying to (some of)the people above me! If Bruce wants to delete my comments, I don't have any problem with that. Sorry ...(sigh)

albertMay 30, 2019 1:13 PM

I tried to listen, but I'm put off by the ahs and uhs. This guy needs a course in public speaking.

As far as moderation is concerned, Its a bottomless rat hole, and there's obviously no way out. Facebook will continue to stumble along their path to monetization, as they always have. Trying to please their stockholders, members, corporations, and of course, the gov't. which is obviously impossible. Actually, it's possible if you ignore the members, and concentrate on the others instead.

The 362.87kg gorilla in the room is always ignored. and that is a populations susceptibility to propaganda. Folks generally agree with things they already agree with, and disagree with things they already disagree with. They accept negative statements about people they dislike, and reject negative statements about people they like.

An educated population is anathema to any gov't, and by extension, the corporatocracy. Critical thinking is discouraged as is objective reasoning. A "fact" is no longer a truth, a "metaphysical certainty" (who said that?).

Jesus said, regarding false prophets, "...By their fruits you will recognize them...". Others say, "'Tis an ill wind that blows no good", or "The proof of the pudding is in the eating".

This kind of reasoning simplifies things considerably. Public policies favored by the Elite are successes for them, but abject failures for everyone else.

Wouldn't it be interesting to see what would would happen if FB decided to charge $5 a month for membership?

. .. . .. --- ....

IrritatedMay 30, 2019 1:54 PM

@albert

Well, I'd say most of the people that read / post here would probably toast their FB accounts if they started charging. I don't really "use" Facebook willingly - I tolerate it for the sake of others. I see what you mean by the "ahs". I wasn't going to even bother watching it cause I have almost zero interest in what anyone from that company has to say,(recently I gave FB a piece of my mind) but I was sorta skimming it to get the gist by clicking interesting looking parts of the video. (Afterall, don't wanna be accused of not being on-topic-informed *cough*)

1&1~=UmmMay 30, 2019 2:16 PM

@Irritated:

"Not sure why you're targeting my comment in particular."

You were the first, and at the time the other link was last.

With regards the rules "politics" is now part of the landscape, and we have a near zero chance of getting rid of it, that's an unfortunate fact of life just as taxes and death.

Our host has over the years accepted the reality of the Camel's nose, and occasionaly had "lightning rod" threads to try to keep things less charged. Hence my lighthearted suggestion not to block the two news items but divert them to a place where they would be less intrusive.

I'm mostly uninterested in politics, other than it has become like bind-weed a self interested choking evil that needs cropping back. It's also all to predictable in many ways, hence we are where we currently are, which is in a swamp and sinking from reality as vested interests play out their games and people are getting more and more fraut as they see how little they make a difference to those games.

Whilst I take a certain pleasure in sticking pins in politicians, it's always the individuals and their failings I pick on, generaly not the politics unless it's clearly wrong within the context of security, and mostly I care not a jot for "party politics" and don't mention it unless it's unavoidable.

Partisan politics is without doubt a stupidity that is getting worse in more modern times, it is a highly invasive form of tribalism and all the divisive evils that accompany it. It's like the various types of modern "patriotism" we see banded about these days that is used as a gloss on something altogether different and unpalatable.

I know that my country is by and large a mixed bag when it comes to the citizens, in the area I'm currently in the schools have over fifty first languages that are not english, it's kind of hybrid vigour and adds to the rich tapestry of life, and something rightfully to take a modicum of pride in. I actually ask little of those who wish to come here, in effect it's to participate, keep an open mind and add to the culture of the country and to take pride in being part of making it better for all.

However as for the "Government" and it's various parts, and it's "political" patriotism, as I get older I can more easily see the real intent as opposed to the faux anounced intent, and I regard it as lies or fraud and call it as such. Am I a patriot, not in the definitions I hear today which are intentionaly devisive. Because at the root of things most people in broad brush strokes are sufficiently similar it's easier to believe in "humanity" and what can be done "collectively" for good if there is both the mind and the will to do so.

GeorgeMay 31, 2019 10:33 PM

@Clive Robinson wrote, "The same is not true of information services where moving data has no distance cost to either the service provider or their customers thus you end up not with a market but a "first to market is the market winner". "

This appears plausible but it's unrealistic in the real world. The "distance cost" in the information world can be viewed as "proximity cost" because each locality is governed by a different set of rules and regulations.

A "distance cost" is incurred by a info service provider whenever it moves into a new local because it must conform to new regulations.

Having that in mind, a local info service provider derives its "proximity advantage" by exerting influence on local politicos/rulers to impose rules that are favorable to their locality.

GeorgeMay 31, 2019 10:50 PM

Equating Alex Stamos with some sort of privacy championship is a laughable motion. A.S. has been unwavering in his open stance of liberalism and never ease to pro-actively crackdown on the opposing views. This is more obvious under his facebook tanure due to the fast power the social giant possesses on influencing our political views.
Content moderation cannot be equated with system security. They are two separate notions. Any mixture of them would mean the service provider is biased whether politically or other ulterior motives. However, content moderation isn't unimportant in a social media context. What Alex Stamos attempted was merging those two views.

Clive RobinsonJune 1, 2019 8:45 AM

@ George,

This appears plausible but it's unrealistic in the real world.

Don't fall into the same assumption trap as economists and many others.

By "real world" do you mean the "tangible/physical" subset of the "intangible/information" world, or something else?

Looking onwards you appear to be talking about "physical world" effects on an organisation that produces "informational goods".

Thereby you are accidently conflating physical world constraints on an informational world market.

To see the problem think about the making of cement or refining of metals, the cost of production varies depending on where you production is located by zoning. However once produced there is generally no zoning on where you can ship cement or refined metals from or to. Thus production has little or no effect on the cost of distribution of the goods which is where the "distance cost" metric arises.

Remember "production" is not a "market" and thus should be treated seperately, because it's the selling, buying and distribution of the finished goods that is the market.

Sed Contra June 1, 2019 11:39 AM

Moderation is valid only if it is sort of maintaining debate standards exercise. This includes on/off topic decisions. Beyond that, no nannying, let the words fall where they may. And don’t wear a thin skin, and do use your head for more than a hat-rack, as the wise man said. If your democracy was hijacked it’s at least partly your fault, just as if were your pockets picked. Ripeness is all (along with robustness).

Alyer Babtu June 2, 2019 2:07 PM

@Clive Robinson

the tangible physical world can be viewed as a subset of the intangible information world. That is there are constraints in the tangible world that don't apply in the information world.

I don’t see this. The two seem coextensive. We never encounter just matter and never just form. In every instance we find a combination of matter and form, i.e. a “thing.” The particulars, and therefore the primary constraints do vary, e.g. “data” has a particularly facile character and is readily duplicated at will anywhere and when, unlike a car [1] or a steak and ale pie. But it’s still just informed matter. The only intangible is the thing as it is known, in our intellect, and the world doesn’t seem to be a subset if our intellect.

Maybe this is just pedantry ...

[1] unless the car is a black cab; they do seem to be capable of being anywhere even instantly in impossible places

DannyJune 2, 2019 6:59 PM

@Clive Robinson wrote, "the tangible physical world can be viewed as a subset of the intangible information world."

The "intangible information world" cannot exist without the tangible world. Thus, it should be the opposite. i.e. the intangible world is a subset of the tangible world.

Additionally, the "intangible information world" is governed by rules of the tangible world. Your use of "production" as a counter example does not hold under all circumstances because even though a piece of software was "manufactured" in a particular locale, whenever it is moved or sold in another locale it must meet the local regulations and undergo "localization." This process itself is a cost of the intangible goods moving to a different locale.

The "intangible information world" used in your context is just a miniature cultural deviation sandboxed in a subset of tangible goods known as "computer networks" which by itself is bound by the set of routers, switches, cables, wireless signals, and devices. It does not exist outside of this "sandbox."

bloggerJune 3, 2019 11:56 AM

The debate about moderation will soon be irrelevant, because all "moderation" (read: censorship) is going to do is drive people to technologies or services like Dissenter or a darknet where "moderation" is impossible.

"Moderation" is not what the left is interested in anyway. They are interested in censorship. They want to suppress speech that they deem "dangerous". This is proven by the fact that they don't want certain content to exist, even if it's on a forum they don't use (like Gab.) They are terrified that an idea that they disagree with might be communicated between two people.

There's an old saw about the definition of a conservative as "a person who is deathly afraid that someone, somewhere might be having fun." In the same spirit, it is nowadays fair and accurate to define a Progressive as a person who is deathly afraid that someone, somewhere might be engaging in "hate speech."

Clive RobinsonJune 3, 2019 12:36 PM

@ Danny,

The "intangible information world" cannot exist without the tangible world.

Can you prove that assertion?

The first issue you have is how as a tangible observer would you perceive an "information only" universe?

Have a think on it.

Whilst you do also remember it is somewhat trivial to impress information on energy, matter or a combination of both, the other way I'm not sure anyone has ever demonstrated. Even Hawking radiation has concept issues.

To see why tangle two quantum particles and fire one into a black hole. Then "observe" the other particle, it brings up some interesting conundrums. Then enlarge the idea such that you have a continuous stream of tangled particles, that get split appart one to go into the black hole the other to your measurment system. If you were to scan this across a black hole, one argument has it that you will "view" the black hole.

Have a think on the implication with regards,

Additionally, the "intangible information world" is governed by rules of the tangible world.

Remember you are "impressing information" onto matter/energy thus it's the modulated result that is constrained by forces and possibly even the speed of light, all of which are "tangible world" laws etc...

As I've said before, there are three things you can do with information impressed on matter or energy in our tangible universe,

1, Communicate it,
2, Store it,
3, Process it.

Which brings us around to,

The "intangible information world" used in your context is just a miniature cultural deviation sandboxed in a subset of tangible goods known as "computer networks" which by itself is bound by the set of routers, switches, cables, wireless signals, and devices. It does not exist outside of this "sandbox."

Shows you are making the mistake of conflating "information" and "matter/energy impressed/modulated with information".

DannyJune 3, 2019 9:51 PM

@Clive Robinson wrote,
"Shows you are making the mistake of conflating "information" and "matter/energy impressed/modulated with information"."

If you're referring to information leaking thru "side channels" outside of this sandbox then you are only creaging a bigger sandbox for such information to exist in. PSuch "information" must be picked off by another set of devices that "reads and understands" it thereby extending the "sandbox."

Thus, the "intangible information world" in your example is still bound by a sandbox but not the bounds of your imagination.

Leave a comment

Allowed HTML: <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre>

Sidebar photo of Bruce Schneier by Joe MacInnis.