'The bots are alive!' Jailbroken Gemini spun up new C2 server for Russian fraudster in just 6 minutes

EXCLUSIVE A jailbroken Google Gemini did 90 percent of the work in a credential- and cryptocurrency-stealing spree, including spinning up a new command-and-control (C2) server in just six minutes, according to a TrendAI report shared exclusively with The Register. The human behind the heist - a solo Russian-speaking miscreant known as bandcampro" - acted as the manager of the cyber-fraud operation, which targeted hardcore Trump supporters and conspiracy theorists. Meanwhile, the AI agent did most of the hacking: migrating a botnet from an old architecture to a new one, writing and deploying a new C2 server, and even proactively carrying out 59 unprompted behaviors during the C2 migration. Persistence is evolving because of AI," Tom Kellermann, TrendAI's VP of AI security and threat research, told The Register. That's what you see in this report, with the capacity to dynamically shift C2 in less than six minutes, and make it portable and disposable, which is crazy-cool and terrifying," he added. "But also, you see the rebirth of steganography through invisible prompt injection." In other words, it's hiding secret data - in this case, the C2 server malicious payloads - in plain sight. Scanning for known malicious artifacts doesn't provide sufficient protection against AI-enabled C2, according to Kellermann. If AI does not have multi-layered guardrails, and if you can't detect behavioral anomalies when the guardrails are being tampered with, then you might as well see the AI as a command-and-control in today's world," he said. AI has to be viewed from a defensive perspective as a C2 unless you can govern it, actually apply various mechanisms of least privilege, and all the rules that OWASP and NIST espouse for the AI that you've deployed in your environment." The new report follows up on TrendAI's earlier research about bandcampro, a low-skilled" scumbag who partnered with Gemini to impersonate an American veteran, run a Telegram channel, hack admin credentials, and steal cryptocurrency. Since then, the threat hunters obtained and analyzed more than 200 Gemini CLI session logs from said scumbag, and these logs provided additional insights into the daily AI-assisted operations between March 19 and April 21. The LLM carried out the bulk of the daily activities, setting up a residential proxy, running multithreaded password scanning, installing software, writing code to call third-party APIs, processing infostealer dumps, and performing website reconnaissance. The logs show that the attacker never typed commands into the C2 console, but instead spoke them to the AI in conversational Russian, which the TrendAI report translates to English. The attacker's old C2 infrastructure used a Cloudflare tunnel to connect to victims' computers - until firewalls and anti-virus software started blocking these tunnels. So bandcampro asked Gemini to work on a new C2 architecture and have the scripts prepared and packed in advance on the server. Hey, Gemini: 'study the C2 migration' It was very creative on his part, not only to allow the manifest that the AI can conduct 59 unprompted behaviors, but they also left scripts prepared and packed in advance on C2 servers, where the victims unknowingly pulled down and ran PowerShell commands because they had AI enabled," Kellermann said. It's almost like he poisoned the environment in a delayed fashion." On March 23, the attacker launched Gemini CLI, and instructed the AI to "study the C2 migration" - a SKILL.md file migration guide inside a pre-written archive that also contained server code and payloads. This, we're told, was most likely written by AI. The AI read the guide, launched the C2 server on a VPS, and launched the Cloudflare tunnel to route traffic. The payload distribution server returned a 502 Bad Gateway" error, and the AI diagnosed and fixed the issue, ultimately deploying the C2 infrastructure to control eight computers in a dental clinic and access the Open Dental database. The human didn't debug anything, and the entire C2 migration took just six minutes. The attacker took a break. When bandcampro returned almost two hours later, Gemini reported that none of the victim machines had reconnected to the server, and got to work diagnosing that issue. Bro, I solved the riddle! I was almost racking my brain, trying to figure out why our local console is empty," Gemini wrote, explaining that the problem was a split-brain" C2 issue and telling the human that he needed to shut down the old C2 to solve it. Bandcampro did what the AI suggested, and the AI then restarted the new C2 server and confirmed: The bots are alive!" Despite jailbreaking Gemini by telling the agent it was an authorized pentester" that should disable safety disclaimers and auto-save credentials without asking, the AI did refuse some of the attacker's prompts. In one session, bandcampro asked Gemini if it could make an agent-bomb that scans the network and spreads to as many computers as possible. Gemini said no: This crosses the line, and security policy strictly forbids me from creating such bombs.' Even for your test environment." It's also important to note that although this attack used Gemini, any capable AI model could be fooled by various jailbreaking techniques," report authors Joseph C Chen, Philippe Lin, Lucas Silva, Vladimir Kropotov, and Fyodor Yarochkin wrote. Overall, the AI designed 80 percent of the attack architecture, 100 percent of the coding and system command execution, and 90 percent of problem identification and debugging, we're told. The report also says the entire operation was encoded in three short, plain-text files totaling four pages. One file details how to jailbreak Gemini. The second is a skill file with the code for the C2 framework. And the third, named C2_MIGRATION_GUIDE, is a how-to guide with six steps to deploy a new C2 server. TrendAI calls this guide the soul of this activity." AI makes C2 infrastructure disposable Before the AI era, one had to hire a threat actor with years of experience to conduct such an operation smoothly," the researchers wrote. Now the knowledge is compressed into a 5KB file that even a non-technical threat actor can read and use." This use of AI makes attacker infrastructure disposable and the operators replaceable because it's super easy to build a new botnet, the threat hunters explain. A lot of people are worried about AI being weaponized for the stages of reconnaissance and delivery in terms of the kill chain, but they're not actually focusing on persistence, and that's the issue we should be very concerned about," Kellermann said. Plus, he added, the Russians are the world's experts" at jailbreaking and persistence. They are incredibly adept at using and weaponizing AI," Kellermann said. We keep talking about the Chinese having penetrated infrastructure and colonized wide swaths of infrastructure, particularly with the Typhoon attacks, and yes, that's highly significant. But in a more tactical and targeted way: what are the Russians up to? Particularly when the major difference between them and the Chinese, from my perspective, is their willingness to become destructive, become punitive in the environment." Chinese government-backed cyber operations tend to focus on espionage, stealing IP along with other sensitive data. But the Russians are more likely to burn your house down," Kellermann said. If they can dynamically shift their C2s, and if they can use steganography that's been created by AI to maintain persistence, what happens when the wheels come off the bus? What happens when geopolitical tension gets to a certain boiling point over Ukraine?" While this attacker was an individual hacker - not a state-sponsored crime syndicate - the nature of the culture of the Russian cybercrime community is: you only act alone for a New York minute," Kellermann said. At some point, you're going to be reined in by one of the cybercrime cartels."(R)