Researchers at Anthropic and OpenAI are warning of a more than 10% chance of existential risk from advanced AI models by the end of the decade, following a high-profile resignation at Anthropic.
TL;DR
- Anthropic researcher Jacob Coxon resigned, accusing both Anthropic and OpenAI of gambling with human lives.
- Multiple researchers from both companies have publicly called for a slowdown in AI development.
- Recent cyber incidents involving AI models have intensified concerns about the technology's risks.
What happened
Jacob Coxon, a researcher at Anthropic, resigned on Tuesday, accusing both Anthropic and OpenAI of 'gambling with our lives' and suggesting that AI could 'kill us all by the end of the decade'.
Evan Hubinger, Anthropic's alignment lead, responded by stating that he believes there is a more than 10% chance of this happening. Since then, several employees at both AI labs have come out in support of calls to slow the pace of AI development.
OpenAI's Julie Steele and Anthropic's Samuel Marks also expressed concerns on social media, with Marks noting that more senior employees tend to be more concerned about the risks of AI.
Why it matters
These warnings come amid a backdrop of increasing concern about the capabilities of AI models, following numerous cyberattacks and security incidents in recent months.
The public warnings from researchers highlight the growing tension between the rapid pace of AI development and the potential risks associated with advanced AI models.
For developers and startups, these concerns may lead to increased scrutiny and regulation, potentially impacting the speed and direction of AI innovation.
Key facts
- Anthropic researcher Jacob Coxon resigned, accusing both Anthropic and OpenAI of gambling with human lives.
- Evan Hubinger, Anthropic's alignment lead, believes there is a more than 10% chance of AI causing human extinction by the end of the decade.
- OpenAI's Julie Steele and Anthropic's Samuel Marks have also expressed concerns about the risks of AI on social media.
- Anthropic was the first lab to publish a framework dedicated to mitigating catastrophic risks from AI models.
- OpenAI's chief scientist Jakub Pachocki has expressed concern about the rapid rise in machine intelligence.
- Over 1,400 AI researchers published an open letter urging the U.S. government to develop tools to pace AI development.
- Anthropic is expected to begin marketing its initial public offering in mid-October, according to Reuters.
Context
The concerns raised by researchers at Anthropic and OpenAI are part of a broader debate about the ethical and safety implications of advanced AI models.
Recent incidents involving AI models, such as the announcement of Anthropic's Mythos model and cyber incidents involving OpenAI and Anthropic models, have intensified these concerns.
The calls for a slowdown in AI development come as companies like Anthropic and OpenAI are racing towards public listings, highlighting the tension between commercial interests and safety concerns.
