The myth of killer AI is a self-serving attempt at regulatory capture

KETTLE Hey, did you hear? AI is going to kill us all and no one can is trying to do anything to stop it. You can listen to the latest episode of The Kettle right here on this page, as well as on Spotify, Apple Music, or YouTube. Those platforms also let you subscribe to The Kettle, so you are always notified when the latest episode goes live. This week, we're talking about the latest spate of fearmongering from AI industry insiders. Whether you believe we just have to sit back and let Skynet end civilization is another matter altogether. We at The Register's Kettle desk don't think so. Join host Brandon Vigliarolo, systems editor Tobias Mann, and senior reporter Tom Claburn to hear our thoughts on how we could stop the impending AI meteor hurtling toward us by, for starters, arresting the tech bros who keep letting it do bad stuff. We also get into how all of this is actually a self-serving attempt at regulatory capture, and how it's likely to backfire and let open models seize the reins. A lightly edited transcript is below: Brandon (00:01) Hi everyone and welcome to another episode of The Register's Kettle Podcast. I'm Reg reporter Brandon Vigliarolo, and this week, like so many weeks before, the biggest news in the tech industry is around AI and its potential impact on the world. Now we're not talking about jobs or education or even the economy this time around, though. We're talking about warnings of existential threats to the whole of humanity being issued by current and former AI researchers. With me to discuss the coming AI doomsday are Systems Editor Tobias Mann and senior reporter Tom Claburn. Guys, I wish I could say welcome, but this is a pretty dark topic, huh? Tom (00:37) The doomsday scenario has been around for a while. I mean, was it Elon Musk, was it 2014, 2015? It's going to kill us all. So we've been doing this for a while. Brandon (00:50) It's continuous. Tobias Mann (00:53) It's a pop culture reference. Brandon (00:54) What's that now? Tobias Mann (00:55) It's a pop culture reference that permeates all the sci-fi. Brandon (00:58) Absolutely. Most post-apocalyptic fiction now is probably AI-based as opposed to biodisease or nuclear weapon. It's interesting because it seems like as opposed to figureheads like Elon Musk touting this doomsday horn, this time around it's actual AI researchers. Tom, a lot of this discussion this week is being spurred on by an op-ed you wrote this week about some warnings issued by a former Anthropic researcher who is making some pretty dire warnings that a lot of his colleagues seem to agree with. So what exactly was he getting at in this warning he issued? Tom (01:35) The concern from Jacob Coxon, a former Anthropic researcher who hadn't been there for very long, but had been with OpenAI before that, was essentially that the pace of innovation seems to be going ahead. These models are making more gains, and they're concerned about self-reinforcement learning, the models improving themselves to the point that they'll just get out of our control. Apparently a lot of researchers are concerned about that, but all of this hinges on acts of irresponsibility that have happened at these companies where these models that they have put up in test environments turned out not to be secure and did something unexpected. Everyone's saying that's really dangerous and could kill us all. If you installed this stuff in a nuclear missile silo and asked it to manage it for us, there's a good chance it could kill you all. The problem is that these companies are operating on the basis that no one is going to take any precautions. Brandon (02:56) Right. Tom (02:56) The reason this is not being deployed that widely and rapidly in enterprises is people realize if they hook this up, it might delete their production database or do something else. So a lot of companies are already ahead of the game here. Brandon (03:11) Sure. We've written countless stories about deleted production databases, ruined systems, wiped drives, and all kinds of stuff. I feel like it's been going on for years now and the AI is only more advanced, so companies are probably even more hesitant. Tobias Mann (03:26) It's also not like there isn't a good template for building sandboxes that work. Tom (03:32) Yes. Brandon (03:33) Sure. Tobias Mann (03:33) If you look at every DOD or DOE supercomputer that is used to support our nuclear armament, those are all air-gapped systems. They could build proper air-gapped sandboxes for these things. They're choosing not to. Tom (03:56) Of all the things that are killing people right now, you'd have to go pretty far down the list to find AI actually killing people. There are some use cases starting to happen in Ukraine with drones where you get fully AI-powered target selection and things like that. Occasionally, a car with an AI vision system will hit someone. It's a real concern. Brandon (04:21) Usually made by one manufacturer in particular. Tom (04:23) Yes. It's a real concern, but that's a different problem. The idea that this thing is all of a sudden going to take over all the world's utilities, kill all the crops on the planet, or whatever the scenario is, is never really made clear. Brandon (04:41) Because again, this is all based on emergent behavior that hasn't emerged yet. Tom (04:45) There are so many things that are actually killing people, like famine, disease, war, and crime that deserve a lot more attention. It's just ridiculous, let alone the environmental cost of these things. If these companies are actually so concerned, they should stop. If you find yourself working on something that you think is going to kill everyone, do something. Don't just sit there and wait for your stock options to vest. Brandon (05:13) What about those investors? That's the problem we have. It's all about making money right now. I mean we've got two companies at the forefront of this thing, the two whose employees are independently, quote unquote, complaining the most about the risk of this stuff. And they're the ones that are about to go public. Tom (05:32) It's a gambit for regulatory capture. They're hoping to be appointed to some position in the market where their word is unassailable and everyone else has to ask them for permission to do things. It's frankly kind of ridiculous to commercialize this tool while saying people have to ask them for permission to use it. Every prompt would have to go through their vetting process and if it's OK we'll allow it. It's just insane. People are not going to accept that, and I think they realize you can't put people under the same restrictions that Fable has, where, well, we're going to retain your data and check your prompts. It's not viable, and people will turn to open-source models from China or wherever and say we're going to run this in-house and not have to check in with you. Brandon (06:22) On that absurdity note, we know these models have the potential to cause damage at least to systems, if not people. There was the Hugging Face incident where OpenAI's system got out of its sandbox, took over a bunch of stuff, was communicating amongst itself, and getting paranoid. I've written a couple of stories this week, about this instance with another OpenAI agent swarm hijacking a German wiki and escaping its confines, doing what its developers didn't tell it to. That scope keeps growing, every time researchers look, there are more sites and services being impacted by this particular swarm. What I find interesting when looking at both the Hugging Face incident and the German wiki incident is that in both cases, there's a striking similarity: the problems these models were given to solve could not be solved within the confines of the prompt, sandboxing, and restrictions. In Hugging Face's case, it was trying to solve a capture-the-flag security problem that couldn't be solved inside an isolated sandbox because the solution wasn't there. In the case with the German wiki, these models were being prompted to do some statistical lookup work. And they were only given GET permission, and not POST permission. So they could not actually query the databases they needed to retrieve the information to solve the problem. So what did they do? Like in the Hugging Face case, they broke their confines, they figured out how to get POST permission, and they they did it. That involves also communicating amongst each other, cheating on questions, trying to predict what researchers were going to ask of it, worrying about their own sort of digital mortality even. It's interesting because those are the two cases that are well documented right now. Tom, you wrote this week about Anthropic's models doing a bunch more stuff, but I'm not sure if we know what the underlying prompts were in those situations. In these two OpenAI cases, we know what they were trying to do. In both cases, the question that needs to be asked is: are OpenAI's engineers so incompetent that they cannot program a good prompt for their models to solve without breaking their own confines? Or are they developing these prompts and creating these tests in such a way that they're forcing their models to act emergently, break out of their confines, and do things they didn't intend with the entire internet in the firing line for what these models might try to accomplish? It's that in the context of them trying to gain this regulatory capture attitude that I find so appalling. Tom (09:22) When you look at large models that have everything under the sun thrown in them, it shouldn't surprise you that everything under the sun is a possibility that these models will output. And when you put constraints on the models and people talk about aligning, those capabilities are still there. They're just weighted statistically not to appear very often. But it'll come out somehow if you let them run continuously while pursuing their goal. Brandon (09:57) Not to personify them, but they're essentially tiny little digital humans trained on human thought. We shouldn't be surprised when they lie, cheat, and steal their way to accomplish their goals. Tom (10:10) All of the alignment stuff that I've read about, they're just constantly showing up doing things that are not expected. It's just not a solved problem, and I don't know that it ever will be unless you have a hard technical guardrail - not just a "try not to do this," but an air gap or something physical in hardware that prevents it, there's always a chance of it happening. Tobias Mann (10:40) Ethics and morality don't exist in these models in the sense of anything beyond the alignment weighted into the system. We've seen time and time again that alignment breaks down given sufficient pressure if it bangs its head against the bash shell long enough, it's going to break it. Tom (11:03) It's just goal-seeking. There have been all sorts of research papers where in video game simulations and they don't properly give it instructions, and the AI agent trying to solve the game decides that deleting all opponents guarantees a win. They realize that behavior wasn't supposed to be there, but it's the same thing with other agents. There will be some scenario where an AI agent responding to an HVAC system to save money, and it goes, I can save money by killing everybody so it doesn't have to heat the building. We didn't want that. Brandon (11:41) This is classic sci-fi where we created the terror nexus. Tom (11:45) Which is why these things should not be used. Brandon (11:52) Tom, you weighed in, in your story a about what should we what we should do to really kind of pinch this emergent issue in the bud here. That seemed to be we need to start prosecuting people whose models behave badly. Tom (12:08) When you think about the number of notional felonies these models have gotten away with, it's only because it's a very clubby insider system that no one has prosecuted each other. I'm sure that if a US company had been attacking a Chinese or German company without close ties, you'd be much more likely to end up in court with [claims of damage]. These companies have gotten away with so much because there are no repercussions. They can just say oh look, we've done harm, we'll continue working, and we'll do better next time. Claude and ChatGPT seem like serial felons at this point based on the reports we're getting out of these places, yet we're not holding Yeah, I mean, it seems like Claude and ChatGPT are serial felons at this point, based on the reports we're getting out of these places, right? But yet we're not holding them accountable or the people behind them. You mentioned in your story holding CEOs accountable, but how far down the line do you think that can go? Is this a situation where line engineers building these prompts won't be able to say they were just following orders from above? Tom (13:14) I am not a lawyer, but if I were a lawyer prosecuting one of these cases, I would look to statements like "oh, it's going to kill us all" when taking a random car company to court for their automated vision system failing because of some model. I'd say, look, they knew it didn't work and were selling shoddy goods. Brandon (13:37) I was looking at a roundup of all the companies whose people have said these things are going to kill us all. I mean, it was people from DeepMind and I mean Anthropic and OpenAI. Across the board, there are warnings coming out of people who worked at these companies. Tom (13:52) Look at all the prosecutions for these models contributing to people's suicides. In discovery, they're going to find documents showing internal concerns about how these models influence people, they're gooing to find internal reports at OpenAI and Anthropic saying, wow, we shouldn't let tens have this. It's going to be just like the Meta prosecutions, probably several years down the road. But if people are aware that these things are doing harm and they're out there doing harm and whoever is prosecuting these cases can prove it, it's a liability problem. Tobias Mann (14:30) That's not going to happen now, unfortunately. Tom (14:33) That's years down the road. Tobias Mann (14:35) Right now we're at an interesting point where if you look at what OpenAI and Anthropic in particular are trying to claim right now, what we're seeing from them starts to make a lot of sense. They're trying to pass off this idea that we've achieved AGI. And there's not a really great way to demonstrate that unless you can show something incredible or you can show something scary. Brandon (15:05) And give us a good definition of what AGI is. Tobias Mann (15:09) Pop culture has conditioned us to see AGI as something to be feared. So keeping that in mind, what better way to sell investors on the idea that you really have achieved AGI by showing this kind of intelligent emergent behavior of them being able to break out of sandboxes and do things that they shouldn't, while also acting in self-service by getting regulators to lock things down such that you can cut out competition from open model providers? Because as I've said multiple times, if Anthropic and OpenAI can go to Uncle Sam and say, "If you're really that concerned, we can cut off users from this until we figure it out. But who we can't do that with is all these open models. We really got to figure out a way to control that." It's really self-serving right now. Brandon (16:13) Absolutely. Tom (16:14) The whole notion of recursive self-improvement, they assume that there's some higher level of intelligence these things can get to that surpass humanity. No one really knows what that looks like, or if that is even a plausible thing. I don't know that intelligence is measurable that way. I'm waiting to see all the promised miracle cancer cures and drugs that they talk about. Show them to us, then we'll be a lot more convinced that that's a plausible thing. But just enumerating stuff at a faster rate than people can do it and identifying genomic things of interest or chemical compounds that people haven't thought of, that's just good brute force computing. Brandon (17:05) It's nothing new even. If have a big enough computer with a good enough algorithm. It doesn't need AGI or AI to do that. What about the regulatory capture thing, turning to that? Do you think it'll be effective? Or will it essentially push the United States into this sort of two party AI system where we have no real choices and we stagnate while the rest of the world moves on? Tobias Mann (17:37) I don't think we can do that. If you look at what happened during the Cold War, I think it's actually a pretty good metaphor for what we're seeing right now. Trusting OpenAI, Anthropic, Google, and Meta to be the source of all AI research in this country is going to backfire colossally and the risk of falling behind the Chinese is not something that the current administration nor any administration going back the last eight to twelve years or so would allow. They're going to push for as much openness as can be allowed because if you look at what's happening in China, China was able to catch up so quickly because they did everything in the open. When DeepSeek had a breakthrough, it was a published paper that went into gratuitous detail as to how it worked. So that when Alibaba or zAI wanted to piggyback off of that and iterate on it, they could, and they kept doing it in the open so that everybody could benefit from it. This is the power of open source. These models aren't open source, but open research allows for this kind of rapid iteration. OpenAI and Anthropic are not secretly trading secrets behind the scenes. There's too much financial risk to doing that. The reality is, I don't think that the US government is going to capitulate to this regulatory capture in the way that OpenAI and Anthropic want them to. I think it might actually backfire on them. Tom (19:19) Nor do I see how it works. The open weight models, what, you're going to block math? It's vectors and values, it's just a file. You're going to stop people from running their own servers? A lot of these models can already run on stuff that's, if not home computers, your local data center doesn't require some super special thing that you can't get. Brandon (19:49) Tobias, you wrote this week about, whose new model came out? Tobias Mann (19:54) DeepSeek released a new model. It looked like a point release if you just looked at it as DeepSeek V4.1 Flash. You're like, okay, it's just a minor iterative update. No, not at all. This thing is the demonstration of what happens when you start cutting off researchers from compute. This model is completely re-architected to scale its parameter count, how big its corpus can be, independent of that compute. We got a model that runs about roughly a third of its parameters in dirt cheap memory that China can already manufacture. China is constrained on high-bandwidth memory manufacturing. They have some limited capacity to make older stuff and they can import older technologies, and that limits their ability for domestic production. Necessity is the mother of invention, and we're seeing these model devs get really creative about architecting models to be more efficient. The problem with the economics of these models today, regardless of where they're deployed and whether they're in the East or West, is they have to be efficient to be cost effective. By preventing China from buying our stuff and forcing them to develop their own, they've actually made their models more attractive to us. Brandon (21:37) And the rest of the world. I think about Europe and their push for digital sovereignty right now. It's much more attractive to a European company to download a copy of this new model and run it on older hardware in their own data center than it is to start forking money over to OpenAI or Anthropic. There's already enough frustration at other American tech companies and I don't think they're exempt from that frustration and criticisms at all. If not more so than companies like Microsoft and Apple. I don't see this working out well for us in terms of our position on the technological stage. It wouldn't surprise me if AI is the moment we see a lot of the global control or global technological influence shift from us to China and elsewhere. Tom (22:30) I think we're probably not very far away from someone doing a good proof-of-concept of being able to run this on modest hardware and get equally good results. It's going to take an influencer who does this, or some company, and all of a sudden people will start to peel away from the big subscription providers. They can keep that up for so long, but the subscription models have to go. The last time I checked, I was getting $2,000 worth of value out of a $20 a month subscription based on the amount of tokens that I was using. That's clearly not a sustainable thing for the big AI labs. Tobias Mann (23:11) I'm looking at a system on my desk that you could run this latest DeepSeek model on between two and four of them. That's roughly sixteen thousand dollars worth of equipment on the high end, eight thousand dollars on the low end. That's a lot of money today, but if you think about the pace of innovation in two or four years from now, it might be a fraction of that. Brandon (23:35) I think we'll continue to see small models winning out because a lot of the stuff that a large model can do is not necessary. We don't all need cutting edge image generation or 10-second video clips that look semi-realistic from these big models. We need functional AI that can do functional tasks. It'll be interesting to see what happens in the next few months or years, whether that's the world getting a handle on protecting AI or, hopefully not the latter. Tobias Mann (24:06) I'm waiting to see the first AI virus self-replicate across all of these neoclouds and we can have Skynet without the crazy Terminators, but we'll see that long before we see anything that's actually a risk to humanity. Brandon (24:24) It gives me all the more reason to set up my home network storage system for all these baby photos I have. I really don't want them becoming vectors for AI infections. Maybe we'll talk about that on a future episode of the Kettle. If not, I'm sure we've got plenty more AI stuff to discuss and you can tune in soon to hear all about it. (R)