How we test models
We compare real outputs built from the same published brief. This page explains the generation routes, how to read run records, and the limits of the comparison.
Prompt policy
Each challenge keeps the same published brief across models. System instructions and CLI wrappers can differ by generation route and form part of the method. We do not tune the task brief for individual models. Infrastructure retries and regenerations are recorded separately; an earlier failed attempt remains part of the run history.
The task brief is published on every challenge. Open a task in the arena and select ℹ Details. If you think a brief favors a model, the receipts are right there.
Runs & harness
Single-file tasks ask for one self-contained HTML document, served without manual fixes. A one-shot API run uses one generation request. A subscription CLI run can produce several model responses from one original user prompt when the CLI automatically continues after an output limit. The current Opus 5.5 CLI protocol permits up to two automatic continuations, for three responses, with tools disabled and no manual followups. Recorded route, turn count and continuation count identify that method. Saved historical Fable 5.2 logs include both single-response and automatically continued runs; older entries without that provenance must not be assumed to use one API completion. Thinking-effort variants use the same task brief and are published separately.
The three Godot tasks use a separate agentic pipeline, godforge. The model writes engine code, compiles a WebAssembly export, tests it and fixes issues until the build passes or the 40-turn limit is reached. This iterative method is identified separately from API and subscription CLI runs.
Currently on the stand: 54 challenges, 182 model variants across 26 families, 4,684 artifacts. Counted from the manifest at build, 2026-11-18.
Cost estimation
Every artifact is a single file, so we estimate output tokens from file size and price them at published per-model output rates:
tokens ≈ characters ÷ 4 cost = tokens × published $/1M output
These are estimates, not invoices. Input and reasoning tokens are not counted, so true cost is higher — especially at high thinking effort. Models without published pricing show no cost. Wall-clock generation time is recorded for runs after 2026-07-02 and includes queue and throttling waits. Total estimated output spend across the site so far: $495.
Blind voting
First-time visitors judge blind by default: labels hidden, panes shuffled. Voting triggers the reveal — who you picked, and how the community split. Votes land in a shared database keyed per task; one vote per task per browser, changeable any time. The tally you see is everyone’s votes combined, and it is public on the leaderboard.
Heavy tasks (WebGL, large simulations, Godot exports) start as posters and load all panes on one click — partly to save your laptop, partly method: no output should win votes just by loading first.
Since 2026-07-10 every vote also feeds an Elo rating. The vote records which models were on screen; the winner scores a pairwise win over each shown opponent and a tie counts 0.5 for both. Ratings start at 1500 (K 32 for a variant's first 30 matches, then 16), one vote carries the same total weight whether 2 or 4 panes were compared, and each arena keeps its own ledger. Ratings are recomputed from the full match log on every read, so removing a vote removes its effect too. Earlier votes never recorded the opponent and stay in the raw counts only.
What this is not
This is a community showcase of outputs built from shared briefs, with generation routes and continuation rules that can differ. It does not use pass@k, held-out test sets or statistical-significance tests. It asks: given the same brief, whose output would you actually ship, play, or read? Community votes measure preference, and preference has biases — we mitigate with blind-first judging and fair loading, not by pretending the votes are science.
Changelog
- 2026-11-13Clarified shared briefs versus route-specific instructions, distinguished API one-shot runs from bounded CLI automatic continuations, and noted incomplete historical continuation provenance.
- 2026-07-10Elo ratings: blind votes now record the on-screen matchup and feed a per-arena Elo ledger; leaderboard ranks by rating, raw vote counts remain.
- 2026-07-05Blind-first onboarding, tournament mode, shareable matchup links, fairness posters for heavy tasks.
- 2026-07-03W11 translation brief added to the writing arena; leaderboard page shipped.
- 2026-07-02Wall-clock generation time recorded for all new runs (includes queue and throttling waits).
- 2026-06-28Fable 5 post-ban rerun published as a separate max variant next to the pre-ban originals.
- 2026-06-24Coding arena expanded to 43 tasks; flash-era classics and simulation briefs added.
Disagree with a verdict?
Good. The arena is the argument: open a challenge, judge it blind, and vote.
Open the coding arena