Anthropic's new tokenizer, introduced with Sonnet 5 and Claude Opus 4.7, increases token consumption by up to 73% compared to OpenAI's GPT-5.x, raising costs for users.
TL;DR
- Anthropic's updated tokenizer significantly increases token usage for Claude, affecting AI pricing.
- Playcode's analysis shows Claude's new tokenizer uses 1.73x more tokens than GPT-5.x for a TypeScript file.
- Anthropic offers introductory pricing to offset increased token generation, but costs will rise in 2026.
What happened
Anthropic's latest tokenizer, introduced with Sonnet 5 and Claude Opus 4.7, generates more tokens for the same input compared to previous versions. The company acknowledges that the new tokenizer can create 1.0–1.35× more tokens depending on the content type, according to Anthropic.
Playcode, an AI app building platform, analyzed the impact of Anthropic's new tokenizer and found that processing a 2,888 character TypeScript file with Claude's new tokenizer emits 1.73x more tokens than GPT-5.x's tokenizer and 1.32x more than Claude's old tokenizer. The increase varies by code type, with Rust at 1.58x, JavaScript at 1.52x, and Python at 1.50x.
Anthropic is offering a reduced introductory rate for Sonnet 5—$2/million input tokens and $10/million output tokens through August 31, 2026. After that, the price will increase to $3/million and $15/million, respectively. If Anthropic's list prices were adjusted to be comparable with OpenAI's GPT-5.x baseline, Playcode suggests Opus 4.8's cost would be $7.50/M input and $37.50/M output instead of the published figure of $5/M and $25/M.
Why it matters
The increased token consumption by Anthropic's new tokenizer complicates AI pricing and makes it more challenging for users to predict costs. This can impact developers and startups that rely on Anthropic's models, as they may need to budget for higher expenses.
For investors, this development highlights the importance of considering token efficiency when evaluating AI models. The competitive landscape may shift as users weigh the performance benefits of Anthropic's models against the potential cost increases.
The introduction of the new tokenizer also underscores the need for better tools and methodologies to compare the costs and performance of different AI models. As the AI market evolves, understanding these factors will be crucial for making informed decisions.
Key facts
- Anthropic's new tokenizer increases token generation by 1.0–1.35× for the same input, according to the company.
- Playcode's analysis shows a 2,888 character TypeScript file processed by Claude's new tokenizer emits 1.73x more tokens than GPT-5.x's tokenizer.
- The increase in token usage varies by code type: Rust (1.58x), JavaScript (1.52x), and Python (1.50x).
- Anthropic offers introductory pricing of $2/million input tokens and $10/million output tokens through August 31, 2026.
- After the introductory period, the price will increase to $3/million and $15/million for input and output tokens, respectively.
- Playcode suggests that if Anthropic's list prices were adjusted to be comparable with OpenAI's GPT-5.x baseline, Opus 4.8's cost would be $7.50/M input and $37.50/M output.
- A team from marketing platform Ploy reported that GPT-5.6 finished pages 2.2× faster, cost 27 percent less, and used about half the output tokens compared to Anthropic's Opus 4.8.
- Anthropic did not immediately respond to a request for comment.
Context
Tokenizers are crucial components of large language models (LLMs) as they map text into tokens that the models can process. The number of tokens required to complete a task can vary significantly between models, making it difficult to predict the final cost of using AI services.
The recent changes to Anthropic's tokenizer highlight the ongoing challenges in AI pricing and the need for better tools to compare the costs and performance of different models. As the AI market continues to evolve, understanding these factors will be essential for developers, startups, and investors.
The competitive landscape for AI models is heating up, with companies like OpenAI and Anthropic continually releasing new versions and updates. Users must carefully evaluate the performance and cost implications of these updates to make informed decisions about which models to use.
