Jacob Coxon, who resigned from Anthropic in September over AI safety concerns, told a Stanford audience this week that frontier labs should dedicate no more than 5% of AI chip compute to training to curb existential risks. He also called for coordinated nation-state pressure to slow development and accused labs of prioritizing speed over safety.
TL;DR
- Jacob Coxon, a former Anthropic researcher, urged a 5% cap on AI training compute to reduce existential risks from rapid AI advancement.
- He called for coordinated nation-state activism to pressure labs to slow development and accused them of prioritizing speed over safety.
- Coxon also proposed international compute monitoring and incremental model releases to minimize societal disruption.
What happened
Jacob Coxon, who resigned from Anthropic last September, spoke at Stanford's Institute for Human-Centered AI (HAI) on Monday, advocating for a 5% cap on AI chip compute dedicated to training future models. He argued this would slow development enough to improve alignment science and reduce risks like bioweapons, nanotechnology misuse, and infrastructure hacking.
Coxon criticized Anthropic and OpenAI for accelerating AI development, particularly recursive self-improvement (RSI), which he described as a pathway to uncontrollable systems. He accused the labs of giving only lip service to slowing down to appease employees and avoid mass resignations.
He proposed nation-state coordination to enforce compute limits and establish an international agency—similar to CERN—to monitor chip usage and verify compliance. Coxon also suggested releasing models incrementally to mitigate sudden societal disruptions, such as mass unemployment.
During the event, Coxon acknowledged the challenges of measuring a model's societal impact before release and expressed frustration that his resignation had not yet driven meaningful change. He advised technically-minded students to contribute to AI development at frontier labs within the next year, before RSI potentially takes over.
Why it matters
For AI developers and startups, Coxon's proposal highlights the tension between rapid innovation and safety, particularly as RSI could accelerate capabilities beyond human control. His call for a 5% compute cap directly challenges the current trajectory of frontier labs, which prioritize speed to maintain competitive edges.
For investors, the debate underscores the growing scrutiny on AI safety and the potential for regulatory interventions that could disrupt business models. Coxon's accusations of labs prioritizing financial incentives over safety raise questions about long-term sustainability and public trust in AI technologies.
For policymakers, Coxon's advocacy for nation-state coordination and international compute monitoring could shape future AI governance frameworks. His warnings about existential risks—such as AI-engineered nanotechnology—add urgency to discussions about preemptive regulation.
However, Coxon's proposals face practical hurdles, including the difficulty of measuring alignment progress and the reluctance of labs to self-impose compute limits. His admission that his resignation had limited impact also raises doubts about the feasibility of grassroots activism driving change.
Key facts
- Jacob Coxon resigned from Anthropic in September 2025 over concerns about AI's rapid development and existential risks.
- Coxon proposed limiting AI training compute to 5% of total chip capacity to slow development and improve alignment science.
- He criticized Anthropic and OpenAI for accelerating recursive self-improvement (RSI), which could lead to uncontrollable AI systems.
- Coxon called for nation-state coordination and an international agency—modeled after CERN—to monitor compute usage and enforce limits.
- He suggested releasing AI models incrementally to minimize societal disruptions, such as mass unemployment.
- Coxon accused frontier labs of prioritizing financial incentives over safety, giving only lip service to slowing development.
- He advised technically-minded students to contribute to AI development at frontier labs within the next year before RSI potentially takes over.
- Coxon expressed frustration that his resignation had not yet driven meaningful change in AI safety practices.
Context
The debate over AI safety and governance has intensified as frontier labs like Anthropic and OpenAI accelerate development, particularly around recursive self-improvement (RSI), which could lead to systems that outpace human control.
Calls for compute limits and international oversight mirror broader discussions about AI regulation, including proposals for compute-based governance and alignment science funding.
Coxon's advocacy aligns with growing public and academic concern about existential risks, though his proposals face skepticism from labs focused on rapid innovation and from policymakers wary of stifling technological progress.
