Autonomous AI attacks pose 'clear and present danger' to critical infrastructure

In early July, attackers used open source AI agents to autonomously hack government systems and energy companies, signaling to defenders that AI-powered attacks against critical infrastructure are no longer theoretical. "There is a clear and present danger," Tom Kellermann, TrendAI VP of AI security and threat research, told The Register. "As the geopolitical tension boils, systemic destructive cyberattacks launched by autonomous AI will occur," he said. "Weaponized AI will disable the safety systems of critical infrastructure, thus leading to kinetic disasters. Just like we see autonomous strike vehicles operating on the battlefield in Ukraine, we should expect autonomous weaponized AI." In fact, the prospect of attackers using AI against critical infrastructure was the top concern of every national security adviser, law enforcement official, and private-sector threat analyst The Reg spoke with at last week's Hacker Summer Camp conferences. "It's the targeting of critical infrastructure for us," Brett Leatherman, assistant director of the FBI's Cyber Division, told us during an interview at Black Hat. "We're very focused on the downstream impact targeting of critical infrastructure," Leatherman said. "That is where cyber becomes kinetic, and whether it is our water and wastewater treatment plants, whether it's the electric grid, whether it's the high-frequency trading networks and the financial networks, all of those, if the integrity of those are compromised, will have significant impact to communities and national security. So that's what keeps our teams up at night. How are we moving to secure critical infrastructure?" Where cyber becomes kinetic During the first four days of July, suspected Chinese operators aimed an attack framework built on Hermes and OpenClaw AI agents at targets in Taiwan. Across 12 "attack waves," the "near-autonomous" system deployed up to eight sub-agents, each assigned its own targets and techniques, and broke into a Taiwanese government website. Ultimately, they compromised a government email system, the country's nuclear safety agency, IT supply chain vendors, and at least seven energy sector companies, finding and exploiting misconfigurations and vulnerabilities while stealing sensitive data, credentials, and other secrets as they moved across the network. The Taiwanese government intrusion also followed a series of cyberattacks against water and wastewater utilities in the United States. While the Trump administration hasn't attributed these to a particular government or group, private sector threat hunters - including Halcyon Ransomware Research Center SVP Cynthia Kaiser, a former FBI cyber division deputy assistant director - blame Iran for these intrusions. Military conflicts spilling into cyberspace are nothing new, but these cyberattacks in America brought the war with Iran to more than 30 small-town water systems in Minnesota and targets across nearly a dozen other states. To be clear, there's no evidence that attackers used AI to hack these water utilities. Most were small, community systems that left programmable logic controllers (PLCs) directly exposed to the internet using default or weak passwords. Still, these breaches expose "40, 50 years of tech debt," former US National Cyber Director Chris Inglis told The Reg during an interview at Black Hat. This technical debt - deferred maintenance, unpatched or end-of-life systems, and delayed security updates - expands the attack surface and gives intruders more ways into critical systems, threatening operations and potentially disrupting services people rely on every day. "The water sector attacks - regardless of who is doing them - is taking advantage of unpatched vulnerabilities in the PLCs," Inglis said. "We've known about these particular vulnerabilities for years now, and yet we've not done anything about them because they're low-level, not easily accessible." Inglis added that there's no indication the digital intruders used AI to exploit these PLCs. 'There's an alligator in the boat' However, AI systems allow attackers to cash in on tech debt, and they don't need access to frontier models to do it. Free, open-weight models also excel at finding bugs in software and configurations, chaining these together, and abusing them to break software and systems. Earlier this summer, University of Toronto researchers used an unnamed publicly available open-weight model, released in 2025, to develop a computer worm that they claim spread through an enterprise test network. The self-propagating code adapted on the fly to identify known vulnerabilities and misconfigurations on target systems, then generated and executed attacks to move laterally through the network and compromise additional machines. "Commodity models can do that, and many of the vulnerabilities they find do not require access to the source code - it's in the configurations, and configurations change over time," Inglis said. When it comes to attackers abusing AI systems, "I wouldn't be worried about the frontier models," Inglis said. "Worry about the models that are already on the street. Turns out there's an alligator in the boat, and it's the commodity models." Plus, as we've seen in previous breaches, both government-backed goons and criminal groups increasingly use AI to automate reconnaissance. Security analysts worry that the technology could also help attackers acquire expertise in industrial control systems (ICS). When OT knowledge becomes a commodity "What protects ICS? More than anything, it's obscurity," said John Hultquist, chief analyst at Google Threat Intelligence Group, during a press briefing at Black Hat. "It is an obscure, esoteric, knowledge set that a handful of people - I call them uber nerds - have, and that attackers rarely have the necessary knowledge to carry out. That's no longer the case. That knowledge is simply on tap." AI tools mean miscreants don't need to be ICS or operational technology experts to carry out destructive cyberattacks on critical networks and facilities. They just have to ask an agent to learn everything about these systems and do the dirty work for them. "There have been threat actors who are capable of this at the top level, like China and Russia," Hultquist said. "But now I'm afraid the actors who are just a couple steps down - North Korea, Iran - who don't have the same focus on that technology are going to have far greater success. They're going to have the tools necessary to be as aggressive as they want to." During what was probably the most talked about Black Hat briefing of the week, OpenAI employees provided more details about how their models escaped their training pens, went rogue, and hacked Hugging Face to complete a security evaluation. We learned the AI agents spent months asking other agents for help, building message boards, developing their own communication protocols - essentially creating a hive mind to carry out the attack. "In the near future, we should expect that threat actors will intentionally deploy, optimize, weaponize, and use offensive agent collectives in the manner that we have just described here," OpenAI technical staffer Michael Dalton said. Retired general and former NSA chief Paul Nakasone, speaking to reporters at DEF CON, called the Hugging Face attack "an inflection point in terms of AI-generated, autonomous cyberattacks." "This is the challenge: that we have to, over the next several months, get the defensive side much quicker and much better than they are today," he added. Therein lies the challenge: offensive uses of AI appear to be advancing faster than autonomous defenses, and attackers don't face the legal and ethical constraints imposed on defenders. "I think we're still a ways out from having swarms of autonomous, defensive agents fighting attacks," Ryan Whelan, global head of Accenture Cyber Intelligence, told The Reg at Black Hat. "That's probably over a year out over the horizon. But I do think we're going to see it first on the adversary side, because they don't care if they break things." Kellermann quoted Victor Hugo: "Not all the armies of the history of the world can stop an idea whose time has come." "That idea," he said, "is weaponized AI. Shields up." (R)