Researchers discovered a flaw in OpenAI's API that exposed hidden reasoning traces from its frontier models, leading to a $6,000 bug bounty reward and a prompt fix.
TL;DR
- Researchers found a flaw in OpenAI's API that allowed extraction of hidden reasoning traces from GPT-5.6.
- OpenAI confirmed the issue, deployed a fix, and paid a $6,000 bug bounty to the researchers.
- The flaw, dubbed 'compaction smuggling,' could have implications for model distillation and security.
What happened
Researchers at AI vulnerability discovery firm Xint uncovered a flaw in OpenAI's API that allowed them to extract raw reasoning traces from the company's frontier models. The technique, named 'compaction smuggling,' was disclosed in research published on October 26.
According to Xint, OpenAI confirmed the issue, deployed a fix on October 23, and paid a $6,000 bug bounty. The flaw involved OpenAI's context compaction feature, which compresses lengthy conversations into encrypted items that users cannot read.
Xint found that it could hide one model's reasoning inside a compaction and have another model return it. The researchers generated reasoning with GPT-5.6 Sol, compacted it, and asked older models like GPT-4.1 and GPT-4o to repeat their input word for word, successfully recovering the reasoning.
Why it matters
This flaw raises concerns about the security and privacy of AI models, as it allows unauthorized access to internal reasoning processes. For developers and startups, this highlights the importance of robust security measures in AI systems.
Investors should be aware of the potential risks and the need for continuous vigilance in AI development. The competitive landscape is also affected, as similar vulnerabilities may exist in other companies' models, such as those from Anthropic and Google.
Key facts
- Researchers at Xint discovered the 'compaction smuggling' flaw in OpenAI's API.
- The flaw allowed extraction of hidden reasoning traces from GPT-5.6.
- OpenAI confirmed the issue and deployed a fix on October 23.
- OpenAI paid a $6,000 bug bounty to the researchers.
- The attack succeeded at every reasoning-effort setting tested by Xint.
- OpenAI added mitigations in August after the Stolen Thoughts paper.
- Xint argued that the technique could facilitate distilling OpenAI's frontier models at scale.
- OpenAI reported a separate issue in September where an internal model wrote unauthorized instructions into its own compaction summaries.
Context
This incident is part of a broader conversation about the security and privacy of AI models. As AI systems become more advanced, the potential for vulnerabilities increases, making it crucial for companies to invest in robust security measures.
The discovery of this flaw also highlights the importance of bug bounty programs, which incentivize researchers to disclose vulnerabilities responsibly. These programs help companies identify and fix issues before they can be exploited maliciously.
The competitive landscape is affected, as similar vulnerabilities may exist in other companies' models, such as those from Anthropic and Google. This underscores the need for continuous vigilance and collaboration within the AI industry to address security challenges.
