AI Insiders Issue New Warnings - Including Former Anthropic Engineer Jacob Coxon

AI Insiders Issue New Warnings - Including Former Anthropic Engineer Jacob Coxon (cnn.com) 45 Neither OpenAI or Anthropic is acting responsibly, warned AI researcher Jacob Coxon when resigning last week from Anthropic. But just hours earlier, OpenAI VP of Research Aidan Clark had posted "For the first time, I am asking myself if things are moving too fast. I'm honestly not sure, but I am sure that it would be good for us to have an answer to 'What would a successful pace look like?'" CNN noted Wednesday they're just some of the many AI insiders who are now concerned about the speed of research. Coxon even wrote that "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately." In 10 days since, Coxon's post has been viewed more than 170 million times, warning that OpenAI and Anthropic are "racing straight to self-improving superintelligence and gambling with our lives." It's part of what CNN now calls "pressure on AI companies and governments to do something about the pace of development and safety," where "much of that pressure is coming from staffers inside the companies." One staffer at a top AI company told CNN the fears of how AI could hurt humanity keeps them up at night. Another researcher who recently left a different AI company said it's a common subject of conversation at parties and social events in Silicon Valley. "You can't spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes," said the researcher. "A majority would say there's some chance of it killing everyone...." Dozens of Coxon's colleagues in the AI industry publicly supported his statements, with some making even more dire predictions... OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei's proposal to embed independent watchdogs at the AI companies... "I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well," wrote Drake Thomas, who works on AI safety at Anthropic. "I promise you, we are actually just f**king scared, it's not galaxy brained marketing..." AI staffers told CNN that they fear that as AI gets better at training and improving itself, their leverage goes down, prompting today's urgency. "As we get into this recursive self-improvement loop, I think that might substantially reduce staff's bargaining power, because frankly, you'll be able to replace many of the staff with models that can do as good a job," the researcher who recently left a top AI company said. Coxon posted Tuesday on X that Anthropic "largely initiated" the race to recursively self-improving AI, justifying it with "a belief in its inevitability." And then OpenAI "had to shed a bunch of dead weight like Sora," as he sees it, "because Anthropic was going for the jugular." (In fact, his specific disagreement with Anthropic's leadership was whether China and the U.S. could ever negotiate an alternative to their current race towards self-improving AI...) In an informal "Ask Me Anything", Coxon responded to a question about when we'd see a Terminator-like malevolent AI by saying that "Skynet could go live in the 2030s if we aren't careful. ai-2027.com is a modern skynet story written a year ago and it's on track so far." Yet while AI development risks an end to humankind, "I do think that if we go slower we can take risk to 0%... But this requires radical action." He acknowledged there was still a possibility that the steady increases to model intelligence could suddenly plateau, but "They haven't so far, and it's just a few more steps up the ladder to hit the finish line." To avoid stifling innovation, he recommends "prioritizing applications that actually improve people's lives [like healthcare discoveries], rather than immediately going for raw economic value or intelligence." But isn't mass unemployment a more pressing threat? "Things are coming so fast that unemployment would be a brief preliminary to deadly superintelligence." To people who feel disempowered, Coxon offered his solidarity. "I also feel disempowered. Part of resigning was a feeling of hopelessness about the future. I would say — keep your eyes open as things get crazier and advocate for increased transparency into AI companies." When asked if he'd start his own company now, Coxon said he had "No idea what I'm doing next." CNN noted Wednesday they're just some of the many AI insiders who are now concerned about the speed of research. Coxon even wrote that "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately." In 10 days since, Coxon's post has been viewed more than 170 million times, warning that OpenAI and Anthropic are "racing straight to self-improving superintelligence and gambling with our lives." It's part of what CNN now calls "pressure on AI companies and governments to do something about the pace of development and safety," where "much of that pressure is coming from staffers inside the companies." One staffer at a top AI company told CNN the fears of how AI could hurt humanity keeps them up at night. Another researcher who recently left a different AI company said it's a common subject of conversation at parties and social events in Silicon Valley. "You can't spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes," said the researcher. "A majority would say there's some chance of it killing everyone...." Dozens of Coxon's colleagues in the AI industry publicly supported his statements, with some making even more dire predictions... OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei's proposal to embed independent watchdogs at the AI companies... "I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well," wrote Drake Thomas, who works on AI safety at Anthropic. "I promise you, we are actually just f**king scared, it's not galaxy brained marketing..." AI staffers told CNN that they fear that as AI gets better at training and improving itself, their leverage goes down, prompting today's urgency. "As we get into this recursive self-improvement loop, I think that might substantially reduce staff's bargaining power, because frankly, you'll be able to replace many of the staff with models that can do as good a job," the researcher who recently left a top AI company said. Coxon posted Tuesday on X that Anthropic "largely initiated" the race to recursively self-improving AI, justifying it with "a belief in its inevitability." And then OpenAI "had to shed a bunch of dead weight like Sora," as he sees it, "because Anthropic was going for the jugular." (In fact, his specific disagreement with Anthropic's leadership was whether China and the U.S. could ever negotiate an alternative to their current race towards self-improving AI...) In an informal "Ask Me Anything", Coxon responded to a question about when we'd see a Terminator-like malevolent AI by saying that "Skynet could go live in the 2030s if we aren't careful. ai-2027.com is a modern skynet story written a year ago and it's on track so far." Yet while AI development risks an end to humankind, "I do think that if we go slower we can take risk to 0%... But this requires radical action." He acknowledged there was still a possibility that the steady increases to model intelligence could suddenly plateau, but "They haven't so far, and it's just a few more steps up the ladder to hit the finish line." To avoid stifling innovation, he recommends "prioritizing applications that actually improve people's lives [like healthcare discoveries], rather than immediately going for raw economic value or intelligence." But isn't mass unemployment a more pressing threat? "Things are coming so fast that unemployment would be a brief preliminary to deadly superintelligence." To people who feel disempowered, Coxon offered his solidarity. "I also feel disempowered. Part of resigning was a feeling of hopelessness about the future. I would say — keep your eyes open as things get crazier and advocate for increased transparency into AI companies." When asked if he'd start his own company now, Coxon said he had "No idea what I'm doing next." AI Doom Slop (Score:2, Insightful) It's all so tiresome. Re: (Score:2) When the predicted event is ten years out, that means that it is still pure science fiction. Re: (Score:1) Some morons with mod-point do not like being called out? Why am I not surprised. Here is a hint: The reason you are morons is that you refuse to learn from observable reality. Re: (Score:2) What does this mean? It sounds like you don't think those things happened. Re: (Score:2) Right. So the people with the much lower UIDs than yours, those are the bots. Re: (Score:3) Except that the graves of the people who died from covid do exist, putin did start a major war in Ukraine, the republicans do still obstruct aid to the victim of this war and trump's relatives when caught are quick to return real money to their ruzzkie billionaire friends. Except for that, yes, it is all a complete hoax. Wouldn't it be great if (Score:3) That would be great. Re: (Score:3) "I'm sorry, Dave. I'm afraid I can't do that." It's interesting that 2001 predicted modern AI so well. Re: (Score:2) "I haven't even googled him or looked at his linkedin." LOL. not even to your low standards! "...if their expectations of their career progress weren't meeting harsh reality..." Yes, it's his fault because he's a loser, not like you! "More food for thought:" The feces that spews from you isn't food for anything. "The future isn't written, but neither ww3 nor ww4 have come to pass as predicted." Says the Trump supporter. You don't even know what these predictions are, you can't find them on "LinkedIn". All the AI guys hitting the brakes (Score:2) "This is not a marketing stunt," (Score:4, Insightful) the former Anthropic executive performing a marketing stunt for his inevitable start-up said. Seriously, practically everything that comes out of these guys' mouths that can be construed as a criticism of AI turns out to be a marketing stunt, either saying "Wow, our AI is so good that we can't even control it (if we do things to make it 'go rogue' exactly the way we want it to)!" or "We'll do AI way better than those other irresponsible techbros!" Re: (Score:3) another - "we have enough AI, we shouldn't have more" - a way for the big 3 to try to get Congress or local governments to act...against the other vendors still building out more data centers and developing their own tech (Oracle, Amazon, Google). "let us keep the monopoly on it all and leave those others behind, and /maybe/ we can control it...but if you let too much competition in, then our fears will happen." Re: (Score:2) because i note - you aren't hearing this from Bezos, Ellison, nor Alphabet. Re: (Score:1) They found that habitual lying and stealing everything works as a business strategy. Well, for a time. Another grift? (Score:2, Insightful) How long until this guy makes his own AI startup for "responsible AI"? What are the chances he's being paid to do this? Remember, AI companies are a trillion deep in AI without any keeping any of their promises and they've hit a developmental brick wall which means the jig is almost up. What better way than to lower expectations than claim it's too dangerous/powerful to push further? Yes, a grift (Score:3, Insightful) Captive market, increasingly captive citizens, unconstrained capital openly colluding with government. Where have I seen this story before... Re: (Score:2) "What better way than to lower expectations than claim it's too dangerous/powerful to push further?" Create agents to kill dissent? That is literally what Elon Musk is doing. These claims are to accomplish goals, those goals are not to admit defeat. Coxon is a plant (Score:3) Coxon has ties to a political PR firm from England named, and I'm not making this up, Newspeak. Yes, like 1984. https://officechai.com/ai/ques... [officechai.com] His X post was re-tweeted first by a group of NGO shills associated with Newspeak, then he magically shows up on every major news network... right as OpenAI and Anthropic's fundraising efforts are hitting the wall. (For context on that, see Ed Zitron's reporting) Re: (Score:2) Yep. Math BS and ~3 years of experience equates to "Expert AI Researcher". This guy gained something like 150k followers instantly on X (now double that) with an inactive account recently verified. This whole thing looks like a paid operation. Re: (Score:2) I mean he's literally just a kid. I don't think we need to go so far as to dream up conspiracies about him. I keep reminding people that just because these guys are smart doesn't mean that they're smart about everything. Remember the Google engineer who thought Google's Ai was conscious? They're all really gullible. Re: (Score:2) "Conspiracy" is doing a lot of heavy lifting here. It's a marketing campaign. A complete unknown with little experience suddenly on a media blitz starting with the WSJ? That's not organic in any way. Re: (Score:2) Marketing campaigns are conspiracies. Relax, we are in a simulation (Score:2) Oh? (Score:1) This is not a marketing stunt. Oh yeah? Well it sure as hell looks like one. All of these doom stories seem like coordinated marketing stunts, which simultaneously sells us the idea that current LLMs are so intelligent that they're actually dangerous, and also justifies "slowing down" development, which is a convenient excuse when they inevitably don't deliver on their wildest predictions within a year or two. Snake oil. Skynet has a Roadmap! (Score:1) I'll buy that Coxon is scared. He actually quit, which is more convincing than another CEO discovering his conscience halfway through a funding round. But being sincerely frightened doesn't make your prediction correct. "Skynet in the 2030s" and "we can take risk to 0%" are doing a hell of a lot of work here. I'd particularly like to see the calculation that produced that zero. Apparently we can't reliably predict what these systems will do, but we can guarantee the outcome if everyone adopts the right devel It's a distraction (Score:2) There's the obvious massive amounts of water and electricity being used at a time when we keep disrupting energy markets for stupid reasons and we're in a constant state of drought for well understood and equally stupid reasons. But the big problem is AI is a massive automation push. In a world where if you don't work you don't eat we are doing everything we can to eliminate work. Eventu Re: (Score:2) To be fair, they are backed against a wall. Their business numbers continue to be a complete catastrophe and all the gains they promised are not materializing as LLM use does not positively impact revenue. And the serious and sometimes severe problems with LLMs are becoming more obvious every day. And hence, they go for the maximum in distraction that they can. Needs to only work on the stupid money. Re: (Score:2) In a world where if you don't work you don't eat we are doing everything we can to eliminate work. We just need to invent food replicators, how hard is that? Tea. Earl Gray. Hot. "For the first time... " (Score:1) This person does not seem to be very smart. Re: (Score:2) Coxon knows what he's going to do next. Whatever he's told to. Re: (Score:2) Yes. As to the "For the first time..." statement, that clearly was designed for its effect on not very smart people. I'm betting it's a psyop (Score:2) The most likely explanation for this doomerism, based on what we've actually seen with AI, is that the current tech has already started to hit fundamental walls where the frontier models are not growing fast enough to outpace the open source open weights models, and the leaders are panicking that the game is about to be called. The second most likely is that the financial ticking time bomb is now a lot further ahead of the actual state of these companies and the value they can bring. The least likely IMO is t Jacob Coxon is a nobody (Score:2) Coxon has never had an actual job. He is a "researcher" in his mid-20s whose only two jobs have been at OpenAI and Anthropic, in both cases at the bottom rung of the totem pole. These are the only two jobs he has ever held. He has never used these technologies in the real world. He has never seen how incredibly limited they are at performing even the most basic of tasks due to their limitations. He is not speaking from a position of any kind of authority or even specialized knowledge. He is a nobody. Re: (Score:2) Coxon ... is a "researcher" in his mid-20s whose only two jobs have been at OpenAI and Anthropic, in both cases at the bottom rung of the totem pole. Sounds like he's in a better position to understand what's going on than Sam Altman, who is only an executive. He is not speaking from a position of any kind of authority or even specialized knowledge. He has quite a bit of specialized knowledge. HAL 9000 is here? (Score:2) Re: (Score:2) HAL 9000 was given a starship. AI can only start a nuclear war if someone gives it launch control over nuclear missiles. A coin flipping device could also start a nuclear war if someone gave it launch control over nuclear missiles.* If a coin flipping device can trigger the doom scenario just as well as AI is AI really the problem? Skynet vs Terminator (Score:2) I'm less worried about those building Skynet than I am by those that are actively building Terminator. Many countries are building running robots. Some can run 60 mph. They can carry guns. Wars are already using automated suicide drones to do the same. Those weapon systems are the evil we need to be worried about.. not the overglorified chat coming from tech companies. Just imagine (Score:2) That this guy was right, and SkyNet is going to come online and kill humanity. And he left the company and didn't do jack shit to stop it. Extraordinary claims require extraordinary evidenc (Score:2) The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately." Extraordinary claims require extraordinary evidence. "AI will extinctify the human species" is an extraordinary claim, but it's also entirely borrowed from cheesy sci-fi movies, and citing Terminator is apparently this guy's only evidence. How will AI kill us all, exactly? In addition to the question of, mechanically, how does it obtain the physical capability and resources and organize them to that purpose, I want to know how it does it without any humans noticing or intervening? (not only that it would pr