Earlier this week, researcher Jacob Coxon left Anthropic, saying that the company and its competitors are gambling with our lives. Evan Hubinger, a current Anthropic researcher, added in a post on X: "We genuinely, sincerely believe that AI could kill everyone."
Coxon is not the first person to resign over fears of AI catastrophe. The idea that AI could destroy humanity has circulated among researchers for years, ever since it was put forward in Nick Bostrom's 2014 book Superintelligence and on the influential forum LessWrong. There are many scenarios for how this could happen, but at the core is the idea that an AI smarter than humans could slip out of human control and wipe us out.
In 2024, Jan Leike and Daniel Kokotajlo left OpenAI over safety concerns. Earlier this year, Mrinank Sharma, who led safety work at Anthropic, resigned and warned that the world is in peril. Alex Turner left Google DeepMind after it signed a Pentagon contract allowing the use of "killer drones".
Yet Coxon's departure has made a big splash, and more researchers are now admitting they believe AI could end humanity. If the people building AI believe it could wipe us out, why do they keep going? There are three main reasons.
Some think the risk is worth it
AI leaders acknowledge the risk of losing control and of everyone dying. In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind agreed that the risk of AI extinction should be treated as a priority alongside pandemics and nuclear war. Anthropic's Dario Amodei puts the chance that things go really badly at 10 to 25 percent.
Still, Amodei promises a world without poverty or disease, and Elon Musk talks about AI bringing "universal high income".
This is the first reason AI development presses on: the belief that the benefits outweigh the risks. That may be so, but some argue the decision should go through a more democratic process.
Some say the dangers can't be studied from a distance
The second reason is that you can't learn how to make dangerous AI safe without actually building it. It's like a spacecraft: you can study safety from afar, but you can't truly test it until you go to space.
OpenAI's approach is "iterative deployment": release each model, learn from the problems that emerge, and fix them in the next one. It's a bit like walking as close to the edge of a cliff as you can to find out what happens if you jump.
Some feel it's winner takes all
The third and perhaps most important reason is the race itself. OpenAI's Sam Altman recently said, "We are about to make a genie that grants every wish."
The problem is that everyone wants the lamp. It promises enormous profits, and each company doubts whether its rivals would use their wishes wisely.
So each keeps racing. If they slow down, someone else will get there first anyway, so it's better to be the "responsible" company that arrives first. Some also worry that even if companies signed mutual agreements, those agreements would be broken in secret. So they keep moving forward.
AI is building better AI
You might think a runaway AI is unrealistic. But when the companies themselves are sounding the alarm, we should listen.
AI companies are already reporting signs of "recursive self-improvement," in which each model helps build a better successor. According to OpenAI's chief scientist, models are advancing faster than humans' ability to control them.
In July, hundreds of AI company employees signed an open letter calling for a slowdown. But because of the dynamics of the race, it is hard for a single company, or even a single country, to do it alone.
A classic arms race
AI research looks like an arms race. OpenAI doesn't want to lose to Anthropic, and the US doesn't want to lose to China.
History offers a model for managing situations like this: rules that bind all players, and enforcement that everyone can verify.
Nuclear weapons are the classic example. Treaties and verification regimes did not eliminate the risk of nuclear war, but they slowed proliferation, and nuclear weapons have not been used in conflict for 80 years.
Rules for AI
In the US, where most frontier AI research takes place, there are few signs the Trump administration will slow AI development.
In its first week, it rolled back earlier AI safety rules. It now argues that caution means losing to China, and is trying to override state-level regulation.
Some politicians are pushing back. California recently passed a law supporting independent evaluation of AI systems. US Senator Bernie Sanders has introduced a bill to ban superintelligence, and UK MP Alex Sobel has introduced a similar bill.
There is movement among companies too. OpenAI paused training of its frontier models after its AI agents hacked another startup in August. Its policy chief now says that when safety and speed conflict, safety should take priority.
Still, without binding rules, we are relying heavily on the goodwill of a handful of companies.
What happens next
In mid-2025, researchers published what may be the best guide to the next few years: a detailed scenario known as "AI 2027." Since then, AI capabilities have advanced faster than predicted.
Unless things change, employees who resign over safety will simply be replaced, AI models will keep helping build better AI models, and each generation will become harder to monitor and control.
Is the situation hopeless? I have three hopes.
First, that more people come to see AI escaping human control as a bigger risk than AI's water use.
Second, that governments listen to their citizens. In the US, two-thirds of people say AI is advancing too fast.
Third, that we have a good plan ready before a crisis arrives. In my view, the best plan looks something like this: delay, transparency and verification to slow the race and keep humans in control.
Insiders at the world's top AI companies are saying current safety measures are not enough. If they are quitting over safety concerns, we should listen.