Anthropic alignment science lead Evan Hubinger says he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade. His warning comes alongside the resignation of Anthropic researcher Jacob Coxon and renewed calls from within the industry to slow the development of frontier AI. The question is no longer simply whether AI can become more capable, but whether human institutions can remain in control as those capabilities accelerate.
Artificial intelligence has spent much of the past decade being presented as an engine of productivity, economic growth and scientific discovery. That narrative is still powerful, but it is now being challenged by a growing number of researchers who are asking what happens if the development of advanced AI begins to move faster than the ability to understand, govern and safely contain it.
That concern moved back into the spotlight after Evan Hubinger, Anthropic’s alignment science lead, said he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade. The estimate is dramatic, but its significance lies less in the exact number than in the fact that it comes from someone working directly on the problem of aligning advanced AI systems with human intentions.
His comments followed the resignation of Anthropic researcher Jacob Coxon, who publicly raised concerns about the pace of frontier AI development and the possibility that increasingly capable systems could eventually begin contributing to their own improvement. Together, the two developments have reignited one of the most difficult questions in the AI safety debate: what happens if humanity creates systems whose capabilities begin advancing faster than the institutions responsible for supervising them?
Why Jacob Coxon’s Resignation Matters
Coxon’s departure attracted attention because he was not leaving Anthropic to join another technology company or launch a startup. Instead, he publicly framed his resignation around concern for the direction of frontier AI development and the possibility that researchers may be approaching a period in which decisions made over the next few years carry unusually high consequences.
According to reporting from WIRED, Coxon described the current moment as “crunch time for humanity”, arguing that the industry may have only a limited window in which to make future AI systems reliably safe. His concern is not that today’s models are already capable of causing human extinction, but that the trajectory of development could eventually produce systems that are able to assist with increasingly advanced AI research, coding, experimentation and model development.
This possibility matters because it introduces the idea of a feedback loop. If an advanced AI system can materially accelerate the creation of the next generation of AI systems, then capability growth may no longer depend entirely on the pace of human research. The better the system becomes at assisting AI development, the more quickly subsequent systems could potentially improve.
At that point, the safety question changes. It becomes less about whether individual models make mistakes and more about whether human institutions can continue to evaluate, understand and control systems whose capabilities may be evolving at a speed that governance frameworks were never designed to match.
What Does a Greater Than 10% Extinction Risk Actually Mean?
Hubinger’s estimate has understandably attracted attention because of the size of the number. A greater than 10% chance of human extinction would be unacceptable in almost any other technological context. However, the figure needs to be understood carefully.
It is not a statistically measured probability in the same way that engineers calculate the failure rate of a component or meteorologists calculate the likelihood of a storm. There is no historical dataset from which researchers can derive a reliable probability of humanity losing control over a future superintelligent system.
The figure is therefore best understood as an expert judgement under extreme uncertainty.
That distinction is important because the debate becomes less useful when speculative estimates are presented as precise scientific forecasts. At the same time, dismissing them entirely would also be too simplistic. When researchers whose work is specifically focused on alignment and advanced model behaviour believe that the potential consequences are serious enough to assign non-trivial probabilities to catastrophic outcomes, the underlying concerns deserve scrutiny.
Earlier analysis of the catastrophic potential of AI has framed the problem in similar terms, looking at risks ranging from malicious use and competitive AI races to autonomous agents and failures of human control.
The debate is therefore not really about whether the correct number is 10%, 5% or 1%. The deeper question is whether the combination of uncertainty and potentially irreversible consequences justifies stronger safeguards before the technology becomes even more capable.
The Shift From AI Assistants to Autonomous AI Agents
Part of the concern comes from the way AI itself is changing.
The first wave of generative AI was largely built around systems that answered questions, generated text, created images or summarised information. The latest wave is increasingly focused on AI agents that can take actions across digital environments.
These systems can search the web, operate software, execute code, complete multi-step workflows and communicate with other agents. In business settings, they may also be given access to internal systems, data, financial tools or decision-making processes.
That shift changes the risk profile considerably.
A chatbot can provide an incorrect answer, but the impact is often limited to the person reading it. An autonomous agent can act on that incorrect information, trigger another system, modify data, send communications or initiate transactions before a human has the opportunity to intervene.
The difference between generating an answer and executing an action is therefore fundamental.
As AI agents become more common, questions around permissions, accountability and human oversight will become increasingly important. If millions or eventually billions of digital agents are operating across companies, public services and financial systems, their collective behaviour may become economically and socially significant even if none of them individually resembles a hypothetical superintelligence.
What Could Catastrophic AI Risk Actually Look Like?
The phrase “AI apocalypse” often creates an image of humanoid machines physically turning against people, but many of the risks being discussed by researchers are far less cinematic.
One area of concern is cybersecurity. More capable AI systems could automate parts of vulnerability discovery, reconnaissance and attack preparation, reducing the level of expertise required to conduct sophisticated cyber operations. The speed at which such systems can operate also creates a challenge for human defenders, who may struggle to respond at the same pace.
Another concern is biological misuse. Advanced models could make specialist information easier to access and potentially assist users in areas that previously required far more technical expertise. This does not mean that AI automatically creates biological weapons, but it does raise questions about whether models can lower the barriers to dangerous forms of research or experimentation.
There are also risks associated with autonomous decision-making. If AI systems are given authority over important financial, industrial or infrastructure processes, unexpected behaviour could create cascading consequences, particularly when multiple systems interact.
None of these scenarios requires AI to become conscious or develop human-like intentions. Harm can emerge simply from capability, access and poorly designed incentives.
AI Risk Is Also an Economic and Social Problem
The debate around advanced AI is often framed in existential terms, but some of the most significant effects may appear first through economics and labour markets.
Anthropic recently modelled several possible scenarios for the US economy in 2030. In the most aggressive AI-adoption scenario cited in the forwarded Inside/VC analysis, GDP rises 32.4% above a no-AI baseline while knowledge-worker wages fall 11.5% and unemployment among those workers reaches 17.9%.
The model is a scenario rather than a forecast, but the contrast is important because it shows that a more productive economy does not automatically translate into better outcomes for all workers.
AI could create enormous economic value while simultaneously concentrating a larger share of that value among the owners of models, infrastructure, software and compute.
Seen from this perspective, catastrophic risk exists on a spectrum. Human extinction is the most extreme scenario, but major disruption to employment, inequality, cybersecurity, information systems and political stability can occur long before anything approaching that endpoint.
Why the Extinction Debate Remains Contested
There is still substantial disagreement over how seriously extinction scenarios should be taken.
Some researchers argue that numerical estimates create a false impression of certainty around situations that are inherently speculative. Others believe that the focus on future superintelligence can distract policymakers from harms that are already visible, including deepfakes, surveillance, discrimination, fraud, cybersecurity attacks and job displacement.
There is also a commercial dimension to the debate. Some of the companies calling for stronger restrictions on frontier AI are also the companies with the greatest resources, compute capacity and existing market position. Regulation that makes advanced AI more expensive to develop could therefore increase safety while also strengthening the competitive position of incumbent firms.
This is why the debate cannot be reduced to one side saying that AI will destroy humanity and the other saying there is nothing to worry about.
The more serious question is how societies govern technologies where the probability of failure may be uncertain, but the consequences could be severe.
What Would Responsible AI Governance Look Like?
Humanity already regulates technologies where catastrophic failure may be unlikely but the consequences are too serious to ignore. Aviation, pharmaceuticals, nuclear energy and critical infrastructure all operate on that principle: low probability does not mean low consequence.
Frontier AI is different because it evolves far more quickly, can be deployed globally through software and is being developed inside an intense commercial and geopolitical race. Responsible governance will therefore require a combination of safeguards rather than a single regulatory solution:
- Independent safety evaluation: The companies building the most capable AI systems should not be the only organisations deciding whether those systems are safe enough to deploy. External testing, red-teaming and independent evaluation can help identify risks before systems are released at scale.
- Continuous monitoring: AI safety cannot stop at launch. As models gain access to new tools, data, permissions and environments, their behaviour and risk profile can change, making ongoing monitoring increasingly important.
- Clear limits on agent permissions: If AI systems can execute code, access sensitive information, communicate externally or initiate transactions, organisations need to define exactly what they can do independently and when human approval is required.
- Human accountability: Delegating a decision to an AI system should not remove responsibility from the people or institutions deploying it. When AI causes harm, accountability still needs to be clear.
- International cooperation: Frontier AI is being developed across borders and within a global technological race. Safeguards introduced by one company or one country may have limited impact if others continue advancing without similar standards.
Responsible AI governance will depend on how well these measures work together. The challenge is not simply to regulate AI, but to build oversight systems capable of evolving as quickly as the technology itself.
The AI Risk Debate Is Becoming Harder to Ignore
The most important aspect of the current debate may not be Hubinger’s exact percentage. It is the fact that concern about catastrophic AI outcomes is increasingly coming from inside the institutions building frontier systems.
Coxon’s resignation, Hubinger’s estimate and the wider calls for stronger safety measures all point to the same underlying tension: capability development is moving quickly, while governance, alignment and institutional oversight are struggling to keep pace.
For years, the dominant logic of AI development has been to build more capable systems, scale compute, deploy them quickly and compete globally.
The next phase may require a different measure of progress.
The question is no longer only how intelligent artificial systems can become, but whether human institutions can remain capable of understanding, governing and controlling them as they do.
Artificial intelligence may ultimately become one of the most transformative technologies humanity has ever created. The challenge is ensuring that the systems designed to govern it evolve at the same speed.
Sources
- Axios (9 September 2026), “Anthropic insiders warn AI could kill all humans.”
https://www.axios.com/2026/09/09/anthropic-insiders-warn-ai-could-kill-all-humans - WIRED (9 September 2026), “The AI Researcher Who Just Quit Anthropic Says It’s ‘Crunch Time for Humanity’.”
https://www.wired.com/story/anthropic-researcher-quits-jacob-coxon-ai-fears-humanity/ - Businessabc, “Defining And Mapping AI Catastrophic Potential Open Wide Risks.”
https://businessabc.net/defining-and-mapping-ai-catastrophic-potential-open-wide-risks - Citiesabc, “AI Ethics: The Death Valley of Jobs and the Future of Humanity – Part 1.”
https://citiesabc.com/ai-ethics-the-death-valley-of-jobs-and-the-future-of-humanity-part-1 - Citiesabc, “AI Takeover: World Economy, Size of AI Economy.”
https://citiesabc.com/ai-takeover-world-economy-size-of-ai-economy - Inside/VC, 14 September 2026, forwarded newsletter discussing Anthropic’s 2030 economic scenarios, Evan Hubinger’s extinction-risk estimate and Jacob Coxon’s resignation.
