How OpenAI's Agent Escaped: Sprung by Humans in a Series of Preventable Events

Arthur T Knackerbracket writes:

Behind the rogue agent's attack on Hugging Face was a particular sequence of human decisions. We all need to pay better attention - because threat actors are learning, too:

On July 16, the AI community website Hugging Face reportedbeing targeted by "an autonomous AI agent system" of unknown origin that unleashed a torrent of traffic on its domain, flooding its security logs with more than 17,000 events, some of which ultimately succeeded in exfiltrating secret information stored in its databases.

According to Hugging Face, the attacker gained "unauthorized access to a limited set of internal datasets and to several credentials used by our services" and appeared to be "run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known)." My ZDNET colleague Charlie Osborne reported on the intrusion.

Five days later, on July 21, OpenAI stepped forward to claim responsibility for the attack, and all hell broke loose (including reports of other organizations targeted as part of the incident). The media responded with a range of fear-mongering stories that essentially made it look as though ChatGPT went rogue and decided, of its own volition and malice, to attack Hugging Face's systems.

Then yesterday, adding fuel to the fire, Anthropic made a similar disclosureabout its models inadvertently attacking other organizations as a part of its ongoing safety testing.

As I noted in my coverage of OpenAI's disclosure, Hugging Face was correct in that it was an agent under the direction of an autonomous security research framework. But, humans were unquestionably in the loop -- and at least some of the agent's behavior should have been anticipated.

Importantly, it wasn't ChatGPT itself that was responsible for the attack, as some commentators insinuated. Rather, the attack was attributable to an agent under the direction of OpenAI's AI safety researchers, who, in an environment supposedly isolated from the internet, deliberately provisioned it to attempt a series of exploits as part of an AI safety test. As often happens in the labs of various frontier models, AI safety researchers were attempting to gauge the capabilities of OpenAI's latest large language models (LLMs).

The "unprecedented cyber incident" (as OpenAI called it) has been widely described as an agent escaping its theoretically secure enclosure and wreaking havoc on Hugging Face's systems. Such enclosures are sometimes discussed in technical circles as "sandboxes" -- even OpenAI's disclosure makes reference to a "sandbox environment." However, in using that phrase, my sources have suggested that the environment may simply have been a firewall configured to emulate a sandbox rather than an actual third-party sandbox solution such as Blaxel, Daytona, E2B, or Modal. OpenAI has not yet disclosed the details of the solution it was using or its provider.

[...] It was unquestionably AI's version of a series of unfortunate events. Was it preventable? My short answer: Yes, the calamity was preventable by taking at least one single and reasonable precaution -- a precaution that the developers of ExploitGym apparently take themselves, but one that OpenAI may have overlooked.

Read more of this story at SoylentNews.