Threads Bans Anyone For Mentioning Hitler, Even To Criticize

Quick test: should saying Hitler, not a good guy" cause you to be banned from your social media account? Seems simple enough. But apparently not for Meta, the largest social media company on the planet.

I've talked about the Masnick Impossibility Theorem and the idea that content moderation is impossible to do well at scale. Part of that explains why there will be a near constant stream of mistakes" in content moderation. Sometimes this is because people just disagree over what is proper, and sometimes it's just because the scale part means that mistakes will be made. Obvious, blindingly stupid responses.

Last year, we released the free Moderator Mayhem game, where players act as front line moderators for a social media review site, where they have to make a large number of important decisions under pressure. In one of the rounds, the player is told that a new AI" driven moderation tool is being introduced, which is supposed to help pre-filter decisions.

When the AI moderator is introduced, the game immediately starts tossing up some hilariously mistaken decisions, often based on the sort of fairly obvious errors that a human would catch, but a computer might miss. For example, the AI blocks someone posting a review of a baseball pitcher because of the phrase killer arm" and blocks a review of a chicken restaurant with the name Cock-a-Doodle-Doo." There are many more like that.

When the game first came out, I worried that some of these examples were a bit over the top. I worried that, as AI-driven tools got better, that portion of the game would feel increasingly unrealistic. However, I'm now thinking that this section of the game may have actually been even more accurate than I could have predicted.

Over the last few weeks, Meta's attempt at microblogging, Threads," has been caught up in a few moderation scandals that seem pretty likely to have been caught up by terribly simplistic algorithmic bans. First, there was a story about how Threads was blocking and sometimes suspending users for mentioning the word cracker" or cracker jacks":

Yes, in some contexts, the word cracker" can be seen as a slur. But, most humans looking this over would recognize in context that it was not used that way here.

Then, more recently, Washington Post reporter Drew Harwell found himself suspended for calling out a Washington Post colleague for criticizing Hitler. First WaPo's Amanda Katz had posted something on Threads saying Do... do people know what Hitler did" and got suspended.

Then, Harwell posted about Katz's suspension, mentioning Hitler by saying Threats suspended [Washington Post Opinions Editor Amanda Katz] for this. She tells me, I stand by my views of Hitler. Not a good guy." And Threads suspended him for that as well.

Harwell notes that the suspension happened almost immediately, again suggesting that Threads' algorithmic checker has an auto-suspension for merely mentioning Hitler (which will certainly change the Godwin's Law equation on Threads). This seemed particularly stupid given that the original was in response to Donald Trump literally praising Hitler.

Eventually, the accounts were restored. And, at least on the cracker" case, Instagram/Threads boss Adam Mosseri claimed that the mistake came from human moderators who somehow were not provided the right tools to view the context of how conversations played out."

Sure, that can happen. I've talked in the past about the importance of understanding context, and how many content moderation failures are due to the lack of context. But it seems difficult to see how the largest social media company on the planet wouldn't have tools in place that let you look at I stand by my views of Hitler. Not a good guy" and think you don't have the context to realize that post is probably not hate speech.

That said, some of this may also come down to the constant drumbeat and criticism of Meta over its moderation choices in the past. There's a reason why the company has increasingly said that it doesn't want to be a platform for discussing politics or the latest news (and actively downranks such content in its algorithms).

But also, come on. These kinds of mistakes are the sorts of things you'd expect to see in a brand new startup run by two dudes in a coffee shop, who hacked together some free-off-GitHub code to handle moderation. Not a company worth $1.5 trillion.