Twitter Accounts With AI-Generated Cat Avatars at Center of Turkish Porn Bot Ring

A data scientist has uncovered a Twitter bot network made up of accounts that have unique avatars featuring AI-generated women, anime characters, and cats. They talk incessantly in Turkish about porn and sports betting.

Twitter bots are getting harder to spot. It used to be easier: when you saw a Twitter egg that had a user name followed by a string of random numbers, you knew you were probably dealing with an account run by a bot. Thanks to machine learning, those accounts now have randomly-generated human faces, a well-known problem. But it's not just faces: there's AI-generated cats and anime avatars, too. These accounts look authentic, but they're not.

An independent data scientist and disinformation researcher who goes by "Conspirador Norteno" online highlighted the trend and explained it in a Twitter thread last week, revealing a bot network of thousands of accounts. Meet @Dilde97512368, a Twitter account with a GAN-generated profile pic that is followed by lots of other accounts with GAN-generated pics," they said.

A generative adversarial network, or GAN, is a technique for creating high quality fake images using machine learning. GANs are programs that learn to produce realistic images by studying a large number of inputs, whether that's pictures of cats, people, or anime girls. Once they've gone through thousands of inputs, GANs use what they've learned to generate new images. Refreshing the website Thispersondoesnotexist.com delivers a wealth of convincing photographs of people that have never existed, for example

Many GAN-generated images contain weird artifacts and details inconsistent with real photographs. With a little practice, people can spot details in GAN images that point to their artificial nature: surreal swirls, odd bits in the background, and details like teeth and eyes where they shouldn't be are all common giveaways. Conspirador Norteno noted that some of the pictures in this bot network had those markers, but they can be hard to spot when the picture is reduced to the size of a Twitter avatar.

Conspirador Norteno and their partner "Dr ZQ" discovered a bot network on Twitter of more than 3,000 accounts using GAN generated images. According to them, the accounts were created around January 25 and use a mix of anime avatars, cats, and faces. Some of the images repeat. More than 900 of them are female faces on a blank white background. Around 275 of them are anime avatars and 307 are cats. The accounts mostly follow each other.

So what are these cats, people, and waifus tweeting about? Porongraphy and online sports betting, and mostly in Turkish. This network replies en masse to specific tweets, with dozens or hundreds of members replying to the same tweet," Conspirador Norteno said.

Bot networks joining together to tweet about pornography is a clear violation of Twitter's terms of service. The site's user policies state that you "may not use Twitter's services in a manner intended to artificially amplify or suppress information or engage in behavior that manipulates or disrupts people's experience on Twitter."

Twitter followed through. We suspended a number of accounts for spam and coordination under our platform manipulation and spam policy," a Twitter spokesperson told Motherboard in an email. Using both technology and human review, we proactively tackle attempts to disrupt the public conversation at scale."

As of this writing, many of the accounts are already being suspended, including Dilde97512368, the account Conspirador Norteno called out in their first tweet.

Twitter has dismantled massive bot networks before, including one in Turkey that was promoting the Turkish political partyAKP and its founder President Recep Tayyip Erdoan in June 2020. This also isn't the first big bot network Conspirador Norteno has uncovered. In August 2020, they uncovered a group of Twitter bots that were selling followers to people by the thousands. That bot network also used GAN-generated avatars to make their accounts seem authentic.