'It would be good for the world' to slow down AI sprints, Anthropic says

It would be good for the world" to slow down the pace of AI development, according to a blog post from Anthropic, which this week began the process of going public with a confidential IPO filing. We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," stated a blog post written by Anthropic co-founder (and former Reg scribe) Jack Clark and researcher Marina Favaro. Executing an actual pause would take a negotiation and monitoring effort on par with nuclear accords, including the agreement from all of the frontier AI labs, as well as support from policy makers around the world. Even then, there is a possibility that some will not abide by any restrictions. Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead. A credible pause also has to specify what triggers it, what lifts it, and who adjudicates." Anthropic has been one of the more alarmist organizations when it comes to the growing capabilities of AI, as it's tried to portray itself as the more safety-concerned alternative to OpenAI, where its cofounders originated. One might dismiss this as clever marketing hype - what better way to convince enterprises to drop millions on largely unproven and sporadically reliable technology than claiming it could be so powerful that it might terminate humanity? In addition, Anthropic's recommendation is taking place the same week it beat archrival OpenAI to filing a confidential IPO as it seeks investment from public markets. Last week, the company announced that it reached a $965 billion valuation, making it one of the most valuable companies in the world. It's pretty rich for a nearly trillion-dollar company to tell everybody else to slow down just as it's about to become unstoppable. Yet the headwinds are swirling. US President Donald Trump signed an executive order this week, which in part directs the Treasury Department to establish an AI cybersecurity clearinghouse" that works with the AI industry and critical infrastructure operators to coordinate and deconflict the use of advanced AI tools. And it comes amid growing public backlash to rapid datacenter expansion. Nonetheless, throughout the paper, Anthropic explained how the human role is narrowing" in model development and attempted to make the case that on the current trajectory, models could soon reach a point where they can self-improve and write better versions of themselves without people in the mix. Once human- and AI-authored code quality reach parity, humans will stop writing code entirely, and shift to only reviewing it," the paper's authors write. But if they can't review code as quickly as Claude can generate it, human review will become the bottleneck to AI development." As of May, Claude authored more than 80 percent of the code merged into Anthropic's codebase, up from the low single digits before Claude Code launched in research preview in February 2025. Newer models are also improving on complex tasks faster than before. The length of human tasks that models can reliably complete on their own had been doubling every seven months as measured in March 2025. Now it is closer to every four months, they said. The Claude 3 Opus model released in March 2024 could reliably complete tasks that take humans four minutes to complete. Claude Opus 4.6 can reliably complete tasks that take humans 12 hours, the team wrote. If this trend holds, tasks that take a skilled person days could come into range this year," the paper states. The paper admits that unknown bottlenecks could emerge, which stop the progress that has been made so far, and the next generation of models could see a slowdown in improvement. The authors cite Amdahl's Law, which states that acceleration in one part of the system leads to choke points in other parts. Anthropic has already encountered one signature of Amdahl's law: as we've begun to push more code around the organization, human code review has become a new bottleneck," they wrote. Anthropic said one area where AI consistently underperforms is in taste" or selecting the next step to take, when left unprompted by humans. Without that judgment, Claude is a capable assistant, but not a system that could drive AI progress on its own," the paper states. It is genuinely unclear whether today's training methods and architectures could unlock that capacity." However, the alternative scenario is that the current trend of models getting better more rapidly may hold. In this world, the pace of progress in AI development becomes determined entirely by the availability of compute ... Humans play a substantially diminished role in their development, likely moving most of our effort towards oversight, validation, and verification of an expanding 'virtual lab' run by AI systems," the paper states. (R)