John Schulman, Chief Scientist at Thinking Machines, has highlighted the critical need for an 'outer loop' in recursive self-improvement (RSI) for AI systems. During a recent discussion with Beren Millidge, Charlie O'Neill, and Dwarkesh Patel, Schulman emphasized that distillation, forgetting, and taste are key components in this process.
TL;DR
- Thinking Machines' John Schulman emphasizes the need for an 'outer loop' in recursive self-improvement (RSI) for AI systems.
- Distillation, forgetting, and taste are crucial for effective RSI, according to Schulman.
- The discussion also covered the challenges of generalization, the role of human feedback, and the future of AI research.
What happened
During a 97-minute conversation with Beren Millidge, Charlie O'Neill, and Dwarkesh Patel, John Schulman discussed the technical challenges of achieving recursive self-improvement (RSI) in AI. Schulman, who previously led RLHF work that shipped ChatGPT, is now the Chief Scientist at Thinking Machines, a lab launched in February 2025 with talent from OpenAI, Meta AI, and Mistral AI.
Schulman described the cycle since 2023, where each new frontier model initially feels like artificial general intelligence (AGI) but quickly reveals bottlenecks in judgment and self-checking. He questioned how many more cycles of this are needed before research truly compounds.
O'Neill, who runs model training at Baseten, discussed the potential of a learner on a chip and the assumptions behind the takeoff story in AI research. He compared the last decade's progress to Moore's law, highlighting the need for new objectives that gradient descent on current architectures cannot propose.
Why it matters
For AI/ML developers and startup founders, understanding the challenges of RSI is crucial for advancing AI capabilities. The discussion highlights the importance of distillation, forgetting, and taste in creating more robust AI systems.
For tech investors, the insights from Schulman and O'Neill provide a glimpse into the future of AI research and the potential for significant breakthroughs. The emphasis on human feedback and the role of environments in training AI models offers valuable context for investment decisions.
The competitive angle lies in the race to achieve RSI, with implications for companies like OpenAI, Anthropic, and the Chinese labs. The discussion also raises questions about the economic incentives and the path to a shared hive mind that trains on everyone's deployment.
Key facts
- John Schulman is the Chief Scientist at Thinking Machines, a lab launched in February 2025 with talent from OpenAI, Meta AI, and Mistral AI.
- Beren Millidge is the CTO at Zyphra, and Charlie O'Neill runs model training at Baseten.
- The discussion highlighted the need for an 'outer loop' in RSI, with distillation, forgetting, and taste as key components.
- Schulman emphasized that each new frontier model initially feels like AGI but quickly reveals bottlenecks in judgment and self-checking.
- O'Neill compared the last decade's progress in AI to Moore's law, highlighting the need for new objectives that gradient descent on current architectures cannot propose.
- The conversation also touched on the challenges of generalization, the role of human feedback, and the future of AI research.
- Schulman predicted that a 10x uplift for AI researchers could be achieved in about two years.
- Millidge estimated that full generality for AI systems in white-collar work could take three years, with 80 to 90 percent earlier once companies rebuild around the models.
Context
Recursive self-improvement (RSI) is a concept in AI where systems improve their own capabilities through iterative processes. This discussion among industry experts provides valuable insights into the current state and future direction of AI research.
The involvement of key figures from Thinking Machines, Zyphra, and Baseten underscores the importance of this topic for the AI community. Their perspectives offer a nuanced understanding of the challenges and opportunities in achieving RSI.
The broader AI landscape is characterized by rapid advancements and intense competition. Understanding the technical and economic incentives driving these developments is crucial for stakeholders in the industry.
