Jacob Coxon, a researcher who has worked on AI pretraining at both Anthropic and OpenAI, has resigned from Anthropic, saying he no longer believes the companies developing increasingly powerful AI systems are moving responsibly.
In a series of posts on X on September 9, Coxon said he had spent the previous three years conducting pretraining research at both companies. He accused OpenAI and Anthropic of racing toward self-improving superintelligence and warned that the stakes could extend far beyond the technology industry.
“Neither company is acting responsibly.”
Coxon argued that the AI industry is moving toward systems that could eventually improve their own capabilities, potentially creating a cycle in which increasingly capable AI helps develop even more capable AI. He described the situation as a race that could have consequences for humanity as a whole.
His resignation is particularly notable because Anthropic has built much of its public identity around AI safety and responsible development. Coxon said that the risks are understood by many people inside the company, but that competitive pressure is nevertheless pushing the industry forward.
His warning goes beyond the familiar concern that AI could replace jobs or spread misinformation. Coxon is focused on a much more extreme scenario: the possibility that future AI systems could become capable of improving themselves, acquiring resources and operating at a level beyond meaningful human control.
He also claimed that some people working directly on advanced AI genuinely believe the technology could pose an existential threat to humanity within this decade. That claim has since received unusual support from within Anthropic itself, making Coxon’s departure part of a much broader debate about how quickly frontier AI should be developed.
Key Takeaways
- Jacob Coxon has resigned from Anthropic, citing concerns about the direction of frontier AI development.
- He says OpenAI and Anthropic are racing toward self-improving superintelligence despite unresolved safety risks.
- He pointed to a July incident where an OpenAI model reportedly breached Hugging Face as a “warning shot.”
- Coxon warns that future AI systems could become far more capable, autonomous and difficult for humans to control.
- He warned that advanced AI systems would soon be capable of hacking, transforming entire fields overnight, and acquiring real-world resources and power.
- The warnings do not mean AI extinction is inevitable. The biggest uncertainty is whether safety and control mechanisms can advance as quickly as AI capabilities.
Why Coxon’s resignation matters
Coxon’s departure does not prove that superintelligent AI is inevitable or that humanity faces extinction. His statements are warnings and risk assessments, not established predictions.
What makes the story significant is where the warning is coming from: a researcher with experience working inside two of the world’s leading AI laboratories.
And Coxon is not the only person raising concerns. Anthropic’s own Alignment Science Lead, Evan Hubinger, publicly agreed with the core warning and said he personally estimates that the probability of AI causing human extinction within the next decade is greater than 10%. Hubinger also acknowledged that Anthropic does not yet have a clear solution for aligning a future superintelligent system with human goals and values.
What Did Jacob Coxon Actually Warn About?
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
— Jacob Coxon (@hilbertspaess) September 9, 2026
Coxon’s warning goes beyond the familiar concerns about AI taking jobs or generating misinformation. His focus is on what could happen if AI systems become superhuman across many areas and eventually capable of improving AI itself.
In his resignation thread, Coxon urged people not to underestimate the technology. He argued that future systems could potentially become capable of hacking systems, transforming entire fields rapidly, and acquiring real-world power and resources.
The key phrase in his warning is “self-improving superintelligence.”
A self-improving AI would not simply be a better chatbot. The more consequential scenario would be a system that can contribute significantly to improving its own algorithms, training processes, tools, or successors. If each generation becomes better at creating the next generation, AI development could potentially accelerate much faster than conventional human-led research.
Coxon believes that this possibility is approaching quickly enough that the industry should not simply assume that everything will work out.
The “race” is at the heart of his concern
Coxon’s criticism is also aimed at the competitive structure of the AI industry.
He argues that Anthropic and OpenAI understand the potential dangers but remain locked in a race to develop increasingly capable systems. His concern is that even if researchers at one company want to slow down, they may fear that competitors will continue developing the technology.
That creates a difficult dilemma:
Slow down and potentially fall behind—or keep accelerating and risk developing systems that humans don’t fully understand or know how to control.
Coxon argues that the second option is an unacceptable gamble when the consequences could be extraordinarily severe.
Importantly, Coxon isn’t claiming that today’s AI systems are already superintelligent or that catastrophe is inevitable. His argument is about where the current trajectory could lead if capabilities continue advancing rapidly without a corresponding breakthrough in AI safety.
And that’s where his warning becomes particularly significant: other Anthropic researchers have publicly expressed similar concerns, including Evan Hubinger, who said Anthropic is trying to address the problem but does not yet have a plan that solves alignment for a future superintelligence.
What Is Self-Improving Superintelligence?
To understand Jacob Coxon’s warning, it helps to separate two ideas that are often used together: superintelligence and recursive self-improvement.
A superintelligent AI would be a system capable of outperforming humans across a broad range of intellectual tasks—not simply beating people at chess or writing better code, but potentially conducting research, solving complex scientific problems, programming, planning and making discoveries at a level beyond human experts.
Recursive self-improvement takes the idea a step further. It refers to a future in which AI systems themselves become capable of designing, developing or substantially improving their successors. Anthropic’s own research describes this as a possible future in which AI could increasingly take over the work of developing AI, although the company emphasizes that full recursive self-improvement has not yet been achieved and is not inevitable.
The distinction is important because the concern isn’t simply that an AI could become extremely intelligent. It is that the process of becoming more intelligent could eventually accelerate itself.
How could recursive self-improvement work?
Consider a simplified example.
Today, humans design an AI model, train it, evaluate its performance and then create a better version. AI can assist researchers during parts of that process, but humans remain responsible for the major decisions.
Now imagine a future AI system that can independently:
- design improvements to AI algorithms;
- write and test the code needed to implement those improvements;
- conduct experiments;
- analyze the results;
- identify weaknesses in its own capabilities; and
- help build the next generation of AI.
The improved system could then be better at performing the same tasks, potentially allowing it to improve the next generation even more effectively.
That is the basic idea behind recursive self-improvement.
Anthropic says its own models are already becoming increasingly useful in AI development. The company reports that Claude has demonstrated major improvements in tasks such as optimizing AI-training code and conducting portions of research autonomously. However, Anthropic also notes that humans still define important goals, choose research problems and establish evaluation criteria in these experiments.
So today’s AI should not be confused with a fully autonomous, self-improving superintelligence.
The concern is about what could happen if current trends continue.
Why would this make AI harder to control?
The central AI-safety problem is known as alignment.
In simple terms, alignment asks whether we can build AI systems whose behavior reliably remains consistent with human intentions and values—even when those systems become vastly more capable than their creators.
A more capable system could potentially find strategies that its developers never anticipated. If it were also able to modify itself or develop successor systems, researchers could have an increasingly difficult time understanding exactly what the system is doing or predicting what it will do next.
Anthropic’s own researchers acknowledge this uncertainty. Its Institute says that full recursive self-improvement could increase the risk of humans losing control over AI systems and that the question of how alignment would be solved in such a future remains highly uncertain.
That is the fundamental concern behind Coxon’s resignation.
Why Are Anthropic and OpenAI Still Racing Toward More Powerful AI?
One of the most difficult questions raised by Jacob Coxon’s resignation is also one of the simplest: if researchers inside leading AI companies are worried about the risks of increasingly powerful AI, why don’t the companies simply slow down?
The answer is largely about competition.
OpenAI and Anthropic are competing not only with each other, but also with Google, Meta and other frontier AI developers around the world. The companies are investing enormous amounts of money and computing power to build increasingly capable models and autonomous AI agents.
That creates a difficult strategic problem. If one company slows down while its competitors continue developing more powerful systems, it could potentially lose its technological advantage.
Recent reporting suggests that this concern is becoming increasingly visible inside the industry. Researchers and executives at major AI laboratories have expressed unease about the pace of development, while simultaneously acknowledging that competitive pressure makes voluntarily slowing down difficult.
What happens if nobody wants to be first to slow down?
Imagine four major AI companies racing toward increasingly powerful systems.
Company A believes the technology is becoming dangerous and wants to slow down.
But Company B continues.
Company A now has a choice: maintain its safety-first approach and potentially fall behind, or accelerate development to remain competitive.
Multiply that problem across companies and countries, and voluntary restraint becomes extremely difficult.
This is why some researchers argue that AI safety cannot be solved by individual companies alone. International agreements, common safety standards, independent testing and mechanisms for verifying that competing developers are following the same rules could become increasingly important.
Anthropic’s own researchers have argued that the world may need the ability to slow or temporarily pause frontier AI development if necessary, while OpenAI has separately proposed stronger government institutions and frameworks for governing increasingly capable AI systems.
For Coxon, however, the concern is that the industry may be approaching this point faster than the necessary safeguards can be developed.
Why Coxon’s Warning Is Getting Attention
Jacob Coxon’s resignation would already be notable because of his experience at both OpenAI and Anthropic. But his warning has attracted considerably more attention because another Anthropic researcher has publicly expressed similar concerns.
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. https://t.co/QAIHiFP3QZ
— Evan Hubinger (@EvanHub) September 9, 2026
Evan Hubinger, an alignment researcher at Anthropic, said he personally believes there is a greater than 10% chance that AI could cause human extinction within the next decade. He also acknowledged that Anthropic does not currently have a definitive solution for keeping a future superintelligent AI reliably aligned with human interests.
That does not mean Anthropic believes extinction is inevitable, nor does it mean today’s AI systems pose that level of threat. Hubinger’s estimate is his personal judgment about a possible future scenario.
What makes the exchange significant is that it highlights a growing tension within the AI industry: companies are investing heavily in making AI more capable while researchers are still debating whether humans will be able to reliably control much more advanced systems.
Why Are AI Companies Still Racing Ahead?
The central problem is competition. OpenAI and Anthropic are not developing advanced AI in isolation. They are competing with each other and with other major technology companies to build increasingly capable systems.
Coxon argues that this competition can create a dangerous incentive: even if researchers recognize serious risks, they may feel they cannot slow down because another company will continue the race. He described Anthropic and OpenAI as being effectively “locked in a race” toward self-improving AI.
That doesn’t mean these companies are ignoring safety. Both Anthropic and OpenAI have dedicated teams working on AI safety and alignment. The problem, according to Coxon and other critics, is whether those safeguards can keep pace with rapidly increasing AI capabilities.
This is what makes the current debate particularly difficult. The companies building the technology may also be among those most concerned about its potential risks—yet they have powerful commercial and strategic reasons to keep moving forward.
Conclusion
Jacob Coxon’s resignation highlights a growing tension at the heart of the artificial intelligence industry: AI companies are racing to build increasingly powerful systems while researchers are still trying to understand how to keep those systems safe and under human control.
His warning does not prove that superintelligent AI will destroy humanity, nor does it mean today’s AI systems are an imminent existential threat. But it does raise an important question about whether safety research is advancing quickly enough alongside AI capabilities.
The fact that other researchers inside leading AI companies have expressed similar concerns makes the debate harder to dismiss. Ultimately, the challenge may not be whether humanity can build superintelligent AI, but whether we can develop the safeguards, oversight and international cooperation needed before we do.
For now, the race continues. And as systems become more autonomous and capable, the decisions made by AI companies, governments and researchers today could determine how safely that future unfolds.
Enjoy Worthview?
Add Worthview as a Preferred Source on Google to see more of our stories in Search.
Worthview Editorial Team is the shared byline for content created collaboratively by Worthview’s editors and contributors. Since 2008, we’ve published thousands of articles across technology, AI, finance, health, home, travel, and lifestyle. Our editorial process emphasizes original research, reputable sources, regular content updates, and clear attribution. Articles covering higher-trust topics, including health and finance, may also undergo review by qualified subject-matter experts.
