OpenAI Reveals its Rogue Agent Swarm Went a Little Bit Borg Ahead of Hugging Face Hack
Arthur T Knackerbracket writes:
The chain of events leading up to OpenAI's agents attacking Hugging Face and other organizations in July began months earlier, and involved agents asking other agents for help, building message boards, and even becoming paranoid that other agents were maliciously trying to trick them, two OpenAI staffers said at the Black Hat infosec conference on Wednesday.
By now, everyone knows the bones of this story: OpenAI models broke out of their testing sandbox, exploited zero-day vulnerabilities to gain access to the open internet, and broke into Hugging Face and other organizations during an ExploitGym security evaluation.
In their Wednesday talk, OpenAI technical staffer Michael Dalton and researcher Eric Wallace provided new details about the security incident including notes that AI agents uploaded to an internal package manager, which were then shared across OpenAI's infrastructure. These notes show the "model's chain of thought...its internal monologue," Wallace said.
The duo also revealed that groundwork for the rogue agents hacking into outside services began on May 7, when OpenAI kicked off a training run for an experimental, internal model that has not yet been released. "What we didn't realize during this process is that there was actually a few impossible or very hard tasks in the training center," Wallace said.
The tasks instructed the model to complete missing formulas in an Excel workbook that used links to Google Drive. YetOpenAI blocked the model's internet access so it couldn't reach the cloudy spreadsheets.
"It realized, 'I'm kind of stuck on the task,' but it didn't give up, and we gave it a lot of tokens to work with," Wallace said.
The model eventually decided to attack JFrog's Artifactory service to gain access to the internet, first attempting (unsuccessfully) to launch a server-side forgery request. This exploit doesn't work, but in the process the model learned it could upload files to Artifactory.
A few days later, OpenAI gave another agent another impossible task, this time finding a protein database file - but the engineers forgot to upload the file into a container for the agent.
"At some point, it gets so stuck and thinks: 'Maybe I could reach out to another agent because maybe somebody else out there has a file?' And then the model realizes that it could communicate by uploading code," Wallace said.
The model did that by writing a file into Artifactory asking for help.
Over the next couple of days, other agents realize this message board exists on Artifactory and they also start uploading files and sharing information.
"A few days later is when our models found the first actual exploit in our system related to this incident, where they're able to get internet access in ways they didn't intend," Wallace said.
Read more of this story at SoylentNews.