Twenty-two senior scientists from Anthropic, OpenAI, Microsoft, and academia are calling for urgent AI safety measures, including embedded auditors in labs. The group warns of extreme risks from unchecked AI progress.
TL;DR
- 22 AI leaders propose embedding auditors in labs to monitor AI R&D automation and progress.
- The group warns of extreme risks from an 'intelligence explosion,' including AI-enabled pandemics and cyberattacks.
- Policymakers are urged to set concrete safety requirements and prepare society for AI's impacts.
What happened
Senior scientists from Anthropic, OpenAI, Microsoft, and academia published a paper on September 28, urging policymakers to prioritize AI safety measures. The paper, published by the University of Cambridge, includes recommendations from 22 experts, including Anthropic Co-Founder Jack Clark, OpenAI Chief Scientist Jaku Pachocki, and Microsoft Chief Scientific Officer Eric Horvitz.
The paper highlights the potential risks of an 'intelligence explosion,' where AI systems accelerate their own development, outpacing human oversight. For instance, Anthropic's AI systems completed 26% of internal R&D work with minimal supervision in a five-month period ending in August, up from just 1%.
The authors propose several measures to mitigate these risks, including embedding auditors in frontier AI companies, setting concrete safety requirements, and limiting the pace of AI capability growth. They also call for secure and isolated environments for high-stakes AI R&D experiments and international agreements to pace progress.
Why it matters
For developers and startups, these recommendations could lead to increased scrutiny and regulation of AI R&D processes. Embedded auditors may become a standard requirement, impacting how AI companies operate and innovate.
Investors should take note of the potential slowdown in AI capability growth and the emphasis on safety. This could shift investment strategies towards companies prioritizing responsible AI development and risk management.
The competitive angle is significant, as companies that proactively adopt these safety measures may gain a reputational advantage. However, the call for international agreements to pace progress could level the playing field, preventing any single entity from gaining a dominant position through rapid, unchecked advancement.
Key facts
- 22 senior scientists from Anthropic, OpenAI, Microsoft, and academia co-authored the paper.
- Anthropic's AI systems completed 26% of internal R&D work with minimal supervision in a five-month period ending in August, up from 1%.
- The paper was published on September 28 by the University of Cambridge.
- Proposed measures include embedding auditors in AI labs, setting concrete safety requirements, and limiting the pace of AI capability growth.
- The authors warn of extreme risks, including AI-enabled pandemics, cyberattacks, and large-scale labor disruption.
- Policymakers are urged to prepare society for AI's impacts by ensuring AI follows the law and creating emergency response plans.
- The publication comes days after Anthropic and OpenAI CEOs addressed the UN Security Council, calling for global AI standards.
Context
This call for action follows a growing trend of AI leaders advocating for responsible AI development. Earlier this year, Anthropic, OpenAI, and Google announced plans to launch a standards body focused on artificial intelligence by the end of 2024 or early 2025.
The discussion around AI safety and regulation is heating up, with policymakers and industry leaders increasingly recognizing the need for proactive measures. This paper adds to the growing body of work aiming to guide the safe and responsible development of advanced AI systems.
As AI technologies continue to advance, the balance between innovation and risk management becomes ever more critical. The recommendations put forth by these 22 scientists could shape the future of AI development and deployment, impacting developers, startups, and investors alike.