Anthropic's New Tokenizer Just Broke the One Number Everyone Compares

Take the same 2,888-character TypeScript file and run it through three tokenizers. Claude's new one turns it into 1.73x more tokens than GPT-5.x's tokenizer does, and 1.32x more than Claude's own previous tokenizer. Same characters. Same code. Wildly different token counts. That single fact quietly breaks most of the dollars-per-million-tokens comparisons published this year, including some of the ones on this blog.

The number nobody checks

The tokenizer swap surfaced in The Register's reporting on July 14. Anthropic knows the new tokenizer inflates counts, which is why Sonnet 5 shipped at a cut-rate introductory price rather than its list price. The article is blunt about the mechanism and the fix Anthropic reached for.

Claude's new tokenizer emits 1.73x more tokens than GPT-5.x's tokenizer and 1.32x more than Claude's old tokenizer. — The Register

Why this breaks every price comparison built on $/M tokens

A price-per-million-tokens figure implicitly assumes a token is a stable, comparable unit of text across vendors. It isn't. A tokenizer is a specific vocabulary built from byte-pair merges over a specific training corpus, and different labs make different choices about how aggressively to merge whitespace, punctuation, and code syntax. Two models can advertise identical $/M rates and still cost different amounts for the identical prompt, because one of them needs more tokens to say the same thing.

This is not a rounding error. A model with a 30% less efficient tokenizer needs a roughly 30% cheaper per-token price just to break even on cost-per-task against a competitor. Nobody puts that adjustment on a pricing page. We wrote earlier about what $50 per million tokens actually buys, and even that comparison assumed the token was the right unit to compare in the first place. It might not be, at least not without a tokenizer-efficiency correction next to it.

• Claude Sonnet 5: $2/$10 per M tokens (input/output) through Aug 31, 2026, then $3/$15
• Claude Opus 4.8: $5/$25 per M tokens
• GPT-5.5: roughly $5/$30 per M tokens
• Meta Muse Spark 1.1: $1.25/$4.25 per M tokens, launched July 9 in US public preview
• GLM-5.2: priced roughly 82% below Opus 4.8 on a per-token basis, per market trackers

Every one of those figures is a sticker price denominated in a currency (the token) that means something different at each vendor. Meta and Z.ai's GLM-5.2 look aggressive on paper, and probably still are once you correct for encoding, but the size of the gap is not what the sticker implies until you know how many tokens each model actually needs to finish the same job.

The only honest fix is cost-per-finished-task

This is exactly why we run outputs live instead of quoting a benchmark score or a vendor's own token-efficiency claim. When a model completes a challenge in the arena, we can count the actual output tokens it generated and multiply by its real price, which nets out tokenizer differences because we're measuring the finished artifact, not an assumed token budget. It's the same discipline behind our look at whether higher effort settings actually earn their extra tokens: count what was spent to get the result, not what the price card implies it should cost.

The GLM-5.2 story adds a wrinkle worth naming as opinion, not fact: reports this week put GLM-5.2 at roughly 40% of developer tokens on OpenRouter, part of a broader swing where Chinese models overtook US providers in weekly API call volume back in February. That share number is itself denominated in tokens counted by whatever tokenizer each model uses, so it likely overstates or understates real usage by some amount nobody has published. Token-share stats have the identical blind spot as token-price stats, just aggregated up to a market level instead of a single API call.

None of this means per-token pricing is useless, it's the only number vendors publish before you've run anything. But treat it as a starting estimate, not a comparison you can trust down to the decimal. If you're deciding between models on cost, run the actual workload through each one and compare the bill, not the sticker. Readers who want this same beat in Polish can follow it at nowosci.ai.

Practically: if you're on Claude and haven't checked which tokenizer your integration is using, check now, because the Sonnet 5 introductory pricing that's masking the 30% overhead expires August 31. After that, the same code you're running today gets measurably more expensive to process, with no change to your prompts and no announcement beyond a pricing page update.