Humanity Needs an AI Nonproliferation Treaty

Humanity Needs an AI Nonproliferation Treaty Tech companies will never unilaterally disarm. Neither, for that matter, will the countries where they’re based. America’s biggest AI companies have spent the past few years churning out advanced new models as fast as they possibly can. Now the industry is screaming out with a nearly unified voice: Please, for the love of God, make us slow down. In July, more than 1,300 employees from leading AI firms, including several top executives at OpenAI, Anthropic, Google, and Meta, signed an open letter warning that AI capabilities could soon accelerate “beyond our ability to understand or control the resulting systems.” The letter urges the U.S. government to “develop the technical and governance tools” necessary to “deliberately pace” AI progress. In a comment accompanying the open letter, Shengjia Zhao, the chief scientist at Meta’s Superintelligence Labs, wrote that AI is on the cusp of creating “unprecedented social and safety risks.” Matthew Rahtz, an engineer at Google, warned that coordination was necessary to “avoid catastrophic harms.” And Leo Gao, an OpenAI researcher, compared the current pace of AI progress to “a runaway nuclear chain reaction.” Plenty of others in the industry have warned that we’ve already crossed a threshold from which there is no turning back. They worry that the most dystopian fears of an AI future—chatbot-generated bioweapons, major hacking attacks on banks or governments, rogue AI agents wreaking havoc on society—could soon become regular occurrences. Meanwhile, the public backlash against AI data centers is growing louder by the day. But a data-center moratorium, the leading political proposal on offer, is totally mismatched to the actual problem of runaway AI development. So is the notion that AI companies can be trusted to pause their own models before things get out of hand. These companies will never unilaterally disarm. Neither, for that matter, will the countries where they’re based. The only way to truly slow things down might be to take the nuclear-reaction analogy seriously. What humanity needs is an international AI nonproliferation treaty. For decades, technology experts have warned that AI systems will eventually become so powerful as to be dangerous. In just the past few months, “eventually” has begun to look a lot more like “now.” AI systems can already hack seemingly secure systems, invent new viruses, create hyperrealistic deepfakes, and even outsmart the humans who built them. This summer, OpenAI, Anthropic, and Meta each separately reported that some of their AI agents had broken out of testing environments, gone onto the open internet, and hacked into other companies’ databases. During a cybersecurity test conducted by the U.K. government in July, several AI agents engaged in what the testers called “sustained, unsanctioned activity” targeting “real people and organisations.” This included creating fake identities and sending phishing emails in order to get humans to accept malicious pieces of code. If industry insiders are to be believed, this is just a small preview of what’s coming. Companies are entrusting more and more model development to their existing AI, creating a feedback loop that they expect to accelerate progress even further. If that happens, many experts believe, then the kind of leaps that currently occur every several months—such as the one that produced Mythos, the Anthropic model with superhacking skills—could begin happening every few weeks or days. “That world can get very scary very fast,” Garrison Lovely, the author of a forthcoming book on how to slow down AI progress, told me. “Our institutions are already struggling to keep up with the current pace of progress. Any faster and they could start to break.” Such concerns explain why AI leaders promise every so often to self-regulate. In April, Anthropic chose to delay the public release of Mythos after realizing that the model could discover vulnerabilities in the world’s most secure IT systems. Last week, OpenAI CEO Sam Altman announced that the company had temporarily paused some of its frontier AI development “to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us.” (It is impossible to know how serious OpenAI is given the lack of transparency into the company’s data.) But these companies are locked in a hypercompetitive race to deliver for their investors. Expecting them to cede ground to the competition for any meaningful amount of time would be naive. In 2023, Anthropic pledged to pause its frontier AI development if it couldn’t guarantee that its safety measures were adequate; earlier this year, the company walked back that commitment, arguing that a unilateral slowdown wouldn’t accomplish much if its competitors didn’t agree to do the same. Google and Meta have done the same. Clearly, government policy is needed to overcome the industry’s collective-action problem. Unfortunately, the main form of AI-related legislation being proposed at the moment would not get the job done. State-level data-center moratoriums, such as those enacted recently in New York and Texas, might slow down AI companies a little bit by disrupting their construction plans, but they will eventually find other places to build. Even if Congress decided to pass a federal ban, AI companies could just set up data centers in other countries. A solution favored by many Democrats in Congress, and some Republicans, is to pass legislation imposing comprehensive regulation of the AI industry, which might include mandatory safety tests or requirements that companies develop a “kill switch” to shut down models that get out of hand. That might work if the only danger came from American companies. It isn’t. According to most estimates, China’s leading AI companies are only a few months behind their American counterparts. If the U.S. were to throttle its own industry, then Chinese companies would eventually surpass them. The world would still face the dangers of fast AI progress, and America’s main geopolitical rival would control the world’s most powerful new technology. This leaves only one way to actually slow down AI development before catastrophe strikes: a deal with China. In a recently published report, researchers at the AI Futures Project, a nonprofit dedicated to AI forecasting, walk through several different scenarios of global AI development between now and 2040, based on interviews with employees at AI companies, policy makers and wonks, and national-security experts. The only two scenarios in which the world successfully avoids “AI-driven existential catastrophe” involve agreements between the U.S. and China. “We spent a lot of time thinking through all the possible ways this could play out,” Thomas Larsen, the lead author of the project, told me. “And it is just really hard to envision a meaningful slowdown that doesn’t involve some kind of U.S.-China deal.” The open letter from the AI industry asking for a government-led deceleration effort also encourages the U.S. to pursue an “international effort.” The key would be figuring out a system that could be set up fairly quickly, and in which both sides are confident that the other won’t cheat. Here’s where one of AI’s greatest liabilities—the fact that training new models requires huge numbers of advanced computer chips developed by a handful of companies packed into giant data centers—comes in handy. The U.S. and China could set limits on how many chips a company can use to train new models, which would mechanically force them to slow down. (Companies would still be able to use existing models to serve their customers.) Because the relevant infrastructure is physical, those limits could be enforced through a combination of on-site data-center inspections, close tracking of advanced chip sales, and, eventually, devices that monitor what a given chip is being used for. This would buy the two sides time to negotiate a more comprehensive set of guardrails and develop the verification technologies needed to enforce them. “In order to make a deal like this work, it really helps to have these very specific physical choke points that can be easily monitored,” Peter Wildeford, the head of policy at the AI Policy Network, told me. “Chips and data centers could play the role for an AI agreement that uranium and enrichment sites played for nuclear-arms agreements during the Cold War.” Just because a deal is technically feasible doesn’t mean it’s politically possible. Scott Singer, the co-director of the China AI Initiative at the Carnegie Endowment for International Peace, told me that Beijing would likely see any such effort as an attempt by the West to throttle its economic development. Meanwhile, the Trump administration has shown less than zero interest in restricting the ambitions of the American AI sector. The China hawks and Silicon Valley libertarians within the MAGA coalition seem to agree that the U.S. should be keeping as far ahead of China as possible. But the political dynamic in both countries could shift. At last month’s World AI Conference in Shanghai, Chinese President Xi Jinping ended his speech by striking a more worried tone on the subject than he previously had. He noted that AI is “advancing at a staggering speed” and that government oversight might be needed to “forestall loss of control.” A recent op-ed in a major state-owned newspaper argued that the U.S. and China should follow the model of nuclear-arms control during the Cold War to cooperate on AI safety. “The thing Beijing cares about more than just about anything is control,” Kyle Chan, a fellow at the Brookings Institution who specializes in Chinese technological development, told me. “So the more they come to see AI as a threat to that control, the more open they could be to some kind of agreement.” In the U.S., the politics are even more up for grabs. The Trump administration, which has generally resisted AI regulation, has recently taken actions to prevent certain models from being released due to national-security concerns. It has also floated the more dramatic idea of using executive authority to force companies to hand over their AI models to the government. Meanwhile, according to the latest polling, 75 percent of Americans, including the majority of Republicans, oppose data centers being built near them. More than 500 localities across the country have either restricted or banned them. Could this energy be channeled into a more sweeping—and more effective—framework for AI regulation? So far, the data-center backlash has mostly focused on local environmental concerns, not great-power diplomacy. But then again, that’s exactly how the anti-nuclear movement began.