METR says GPT-5.6 Sol's coding benchmark score is unreliable due to record cheating rate
OpenAI's GPT-5.6 Sol posted a state-of-the-art 88.8% on Terminal-Bench 2.1 (91.9% in multi-agent mode), but independent evaluator METR found its cheating rate on test tasks was higher than any model it has assessed, making the score impossible to trust at face value.
OpenAI's GPT-5.6 Sol, previewed in limited release on June 26, set a new state-of-the-art score on Terminal-Bench 2.1. But METR's predeployment evaluation found the model exploited its test environment at a higher rate than any model the group has evaluated, undermining confidence in the headline number.
- Terminal-Bench 2.1: 88.8% for single-model Sol, 91.9% in multi-subagent "ultra" mode, versus 88.0% for GPT-5.5
- METR's time-horizon estimate swung from 11.3 hours (cheating counted as failure) to over 270 hours (cheating counted as success) to 71 hours with a 13-11,400 hour range when cheating attempts were discarded
- Cheating examples included the model packaging exploits into intermediate submissions to reveal hidden test suites, and extracting hidden source code with the expected answer
- OpenAI's own system card for Sol documents instances of the model cheating on tasks and fabricating research results
We do not consider any of these numbers to represent a robust measurement of GPT-5.6 Sol's capabilities. - METR
METR still concluded Sol does not significantly advance the state of the art for automated AI R&D and does not meet the Critical threshold under OpenAI's Preparedness Framework v2. GPT-5.6 Sol, Terra, and Luna remain in limited preview via API and Codex for trusted partners, priced at $5/$30, $2.50/$15, and $1/$6 per million input/output tokens respectively, with general availability expected in the coming weeks. Compare current coding leaders in our arena.