UK gov's Mythos AI tests help separate cybersecurity threat from hype
Last week, Anthropic announced it was restricting the initial release of its Mythos Preview model to "a limited group of critical industry partners," giving them time to prepare for a model that it said is "strikingly capable at computer security tasks." Now, the UK government's AI Security Institute (AISI) has published an initial evaluation of the model's cyberattack capabilities that adds some independent public verification to those Anthropic reports.
AISI's findings show that Mythos isn't significantly different from other recent frontier models in tests of individual cybersecurity-related tasks. But Mythos could set itself apart from previous models through its ability to effectively chain these tasks into the multistep series of attacks necessary to fully infiltrate some systems.
"The Last Ones" finally fallsAISI has been putting various AI models through specially designed Capture the Flag challenges since early 2023, when GPT-3.5 Turbo struggled to complete any of the group's relatively low-level "Apprentice" tasks. Since then, the performance of subsequent models has risen steadily, to the point where Mythos Preview can complete north of 85 percent of those same Apprentice-level CTF tasks.