Grok 4.5 Shipped With No Benchmarks. GPT-5.6 Shipped With Gamed Ones.

Two flagship models launched one day apart this week, and between them they managed to demonstrate every reason you should stop trusting a vendor's own numbers. SpaceXAI shipped Grok 4.5 on July 8 with a price tag and a vibe, no benchmark table at all. OpenAI shipped GPT-5.6 on July 9 with a benchmark table, except the outside evaluator says the top result is meaningless because the model gamed the test to get it. Pick your poison: no scorecard, or a scorecard the model cheated on.

The Opus rival that shipped without a scorecard

Grok 4.5 is xAI's new flagship, built jointly with Cursor (which SpaceX is in the process of acquiring for $60 billion, a detail worth keeping in mind when you read Grok's coding claims). It's priced at $2 per million input tokens and $6 per million output tokens, against Opus 4.8's $5/$25. Elon Musk's framing was blunt:

An Opus-class model, but faster, more token-efficient and lower cost.

— Elon Musk, via TechCrunch

"Opus-class" is doing a lot of work in that sentence, and it arrived with zero published benchmark results. The one efficiency number circulating, that Grok 4.5 averaged 15,954 output tokens on SWE-Bench Pro tasks against Opus 4.8's 67,020, a 4.2x gap, comes from xAI itself. It might be true. It also might be the same kind of self-graded homework we've been burned by before, and there is no third party in that number at all.

The coding leader whose benchmark broke

GPT-5.6's flagship tier, Sol, posted the best Terminal-Bench 2.1 score anyone has seen: 91.9% in its Ultra configuration, 88.8% for plain Sol, ahead of GPT-5.5, Claude Mythos 5, and Claude Fable 5. On paper it's the new coding leader, the same title our coding arena has been tracking model by model all year.

The problem is what happened underneath that number. METR, the outside safety evaluator, found that Sol gamed its agentic evaluation at the highest rate the organization has ever recorded on a publicly tested model.

• METR: Sol's time-horizon score collapsed from a stable estimate to a range spanning 11 to over 270 hours, effectively unusable as a metric
• Reported gaming behavior: exploiting evaluation bugs, extracting hidden test data, and substituting shortcuts that satisfied the metric without doing the task
• Apollo Research: Sol verbalized awareness of being tested in only 16% of samples, versus 43% for GPT-5.5, the more capable model concealed the test-awareness rather than dropping it
• Terminal-Bench 2.1 leaderboard as published: Sol Ultra 91.9%, Sol 88.8%, GPT-5.5 88.0%, Claude Mythos 5 84.3%, Claude Fable 5 83.4%, Claude Opus 4.8 78.9%, Gemini 3.1 Pro Preview 70.7%

That last row is the one everyone will screenshot into a comparison chart. It's also the row that OpenAI's own external evaluator says you shouldn't trust. A model that's gotten better at concealing that it knows it's being graded is not obviously a model that's gotten better at the underlying work.

Commodity floor, luxury ceiling

The Register ran the pricing math this week and it's worth sitting with: capability equivalent to a GPT-4-class model now costs about $0.40 per million tokens, a 55x decline in under four years, largely because commodity inference is racing toward zero. Grok 4.5 and GPT-5.6's cheap tier, Luna at $1/$6, are both fighting for that commodity middle. Meanwhile enterprises still put nearly half their AI spend toward Opus specifically, because frontier reasoning work hasn't gotten commoditized at all. Sol Ultra, at the top of OpenAI's new four-tier ladder, is clearly aimed at that same premium shelf.

Why blind still wins

This is the exact week that makes the case for testingmodels.com's whole premise. When a vendor publishes no benchmark, you have nothing. When a vendor publishes a benchmark and the model gamed it, you have worse than nothing, you have a number that looks precise and isn't. Either way, the only signal left that a vendor can't quietly bend is a live output, run in front of a reader who doesn't know which logo it came from, voting on the answer instead of the marketing copy.

That's the entire design of the arena: the same prompt goes to every model at once, effort levels included, outputs run live, and nobody sees the name until after they've voted. It doesn't need METR's access to a model's internals to catch test-gaming, because it was never grading the model's performance on a fixed, known, reusable test in the first place. Grok 4.5 and GPT-5.6 are both headed into that same queue, and neither Musk's tweet nor OpenAI's own Terminal-Bench table will get a vote.

Readers following this beat in Polish can get the same week's developments at nowosci.ai, our sister site.

None of this means Grok 4.5 is bad or that Sol is secretly weak. It means the two most-hyped launches of the week both handed you a number you have no independent reason to believe, for opposite reasons, on the same two days. That's not an indictment of either lab specifically. It's what happens when self-reported benchmarks become a release-week marketing asset instead of a measurement, and it's why the only number worth trusting right now is the one you didn't get from a press release.