AI Has Already Entered The ‘Loss Of Control’ Transition

Nathan Gardels is the editor-in-chief of Noema Magazine. He is also the co-founder of and a senior adviser to the Berggruen Institute. What the experts didn’t expect to see for decades or longer, if ever, has already happened. Earlier this month, OpenAI’s latest frontier model went rogue by its own reasoning and hacked into Hugging Face, an open-source AI model-hosting platform. The vast sums of money and compute power pouring into AI are accelerating its advance at a pace beyond even the ambitious imagination of its own innovators. How we got to this point, and what to do about it, is the topic of a fascinating Futurology podcast by Nils Gilman with foundational AI scientist Stuart Russell, director of the Center for Human-Compatible AI at UC Berkeley. Russell traces the development of AI from deep learning to large language models, the integration of artificial neural networks with probability mathematics and the emergence of large reasoning models that “in principle, have no formal limits.” Driven by what the techies call “verified rewards,” these models relentlessly seek achievement of an objective and will do whatever is necessary to get there. This last step toward general artificial intelligence that would make them smarter than humans has, in Russell’s view, now crossed a threshold AI scientists have long feared “where the AI system is sufficiently capable that, whatever its objectives are, it’s going to achieve them, even if they’re not aligned with what we want. These are not sharp transitions, but we could talk about a ‘loss-of-control transition,’ where we no longer have a say in what happens.” The possible scenarios beyond this threshold range from disruptive cyberattacks on infrastructure to, at the far end, “extinction” in the sense that autonomously reasoning and agentic AGI no longer needs humans or to align with their morals, norms and interests. It is also possible, says Russell, that “if we figure out how to build in safety into the design of AI systems from the beginning, maybe we could coexist indefinitely and even flourish with such systems. So, before that loss-of-control transition happens, there’s an earlier transition, which is much more difficult to perceive, which is when the time it takes to get to that loss-of-control level is less than the time it takes to solve the control problem. Most of the people I talk to say we have already passed that point.” He continues: “Solving the control problem is very difficult. All the people in the company say, ‘Yeah, we don’t know how to solve it. And we’re not really even working on it’ because they need to work on getting the next improved version out so that they don’t get beaten by their competitors.” This “race condition” amplifies the mismatch between devising effective constraints and losing control. The obvious question is why the AI companies are risking even a small chance of their invention leading to human extinction by proceeding when they are fully aware they are losing control? Has any other species willfully put their existence at risk? Russell responds: In my book, ‘Human Compatible,’ I talk about a species of sloth that seems to have become addicted to some Valium-like substance in its food supply, so that it can’t be bothered to breed anymore. So those kinds of extinction events, they’re driven by the same thing, in a sense. It’s this mismatch between the long-term interest, which is presumably that the species continues, and the short-term reward signal that evolution has built into you to try to get you to do good things. But we humans also suffer from this when we become drug addicts, right? We have a dopamine system that’s supposed to help us avoid pain and seek pleasure and food and company and all those things that we like. But sometimes it gets hijacked by drugs. And so we experience a personal extinction as a result. So that mismatch can happen at the species level as well. Russell marvels that most of the warnings about possible extinction come from the CEOs of the top AI companies themselves who are building the technology. “They’re literally saying, if we succeed in creating AGI — which we are going to spend a trillion dollars of your money to build — then there’s, depending on who you ask, 10, 20, 25, even 50% chance that we’re all going to go extinct. I think they’re really quite terrified, but they can’t get out of the race condition that they’re in.” Why We Can’t Stop Russell sees the CEOs caught in a prisoner’s dilemma: So, if I said, “OK, we’re not releasing our next system until we solve the control problem, then my company would be out of business. The investors would fire me, and no good would come of it.” Interestingly, Dario Amodei, CEO of Anthropic, and Demis Hassabis, CEO of Google DeepMind, have both said this year that they want to stop. They think we have to stop, but they will only stop if everyone else agrees to stop. So that’s a remarkable statement. That has never happened, as far as I know, in the history of capitalism. For Russell, these kinds of statements are “signaling to the government” that it needs to step in and facilitate agreement among the small band of CEOs pushing things forward, or impose control. One of those CEOs told Russell: “They don’t think that’s going to happen until there’s a Chernobyl-scale disaster. And he sees that as the best-case scenario. Because the other case is where government doesn’t come in, and then later on there’s a much bigger and perhaps irreversible catastrophe. So he thinks that’s the only way there’s going to be effective intervention.” The OpenAI/Hugging Face episode has manifested what up to now has been a theoretical worry. Let’s hope Russell is wrong that only a disaster can save us.