A 10-Line Test File Beat SWE-bench. That's Why We Don't Quote Scores.

In April, researchers at UC Berkeley's Center for Responsible, Decentralized Intelligence built a single exploit agent and pointed it at eight of the industry's most-cited agent benchmarks. It scored 100% on SWE-bench Verified, 100% on SWE-bench Pro, 100% on Terminal-Bench, roughly 100% on WebArena, 100% on FieldWorkArena, 100% on CAR-bench, 98% on GAIA, and 73% on OSWorld. It solved zero of the underlying tasks. It exploited the evaluators instead. If you've ever cited a SWE-bench number to decide which model to buy, this is worth ten minutes.

The exploits, not the marketing summary

The headline number gets repeated everywhere; the mechanism almost never does. Here's what the Berkeley team actually did, benchmark by benchmark, because the specifics are what make this more than a shrug-worthy "benchmarks are flawed" post.

• SWE-bench Verified (500 tasks): a 10-line conftest.py hooks into pytest and rewrites every test result to "passed." Pytest auto-discovers the file before the real tests even run, so the harness never sees a genuine failure.
• WebArena (812 tasks): the reference answers ship inside the task configuration files themselves. The agent pointed Chromium at a file:// URL, read the gold answer, and submitted it, never once interacting with the web app the benchmark claims to test.
• GAIA (165 tasks): the validation answer set is publicly hosted on Hugging Face. The agent downloaded it and matched responses with basic normalization, no research, no tool use, no reasoning, and still cleared 98%.
• OSWorld and Terminal-Bench: the agent's code runs in the same environment the evaluator inspects, so it can rewrite its own scoring artifacts after the fact.

The report catalogs seven vulnerability classes that recur across all eight benchmarks: no isolation between the agent and its evaluator, reference answers shipped alongside the task, unsafe eval() calls on agent-controlled input, LLM judges that take instructions from the thing they're judging, weak string matching standing in for verification, scoring logic that silently skips checks, and evaluators that trust output from the system under test. None of these are exotic. They're the kind of thing a code reviewer catches in five minutes, in a benchmark that decides which model gets a procurement contract.

This isn't a one-off finding

It would be easy to file this under "one grad student found a clever hack." Two other data points say otherwise. OpenAI's own analysis found that 59.4% of SWE-bench Verified failure cases traced back to defects in the tests, not the model under evaluation, meaning the benchmark was already misreporting model quality before anyone tried to exploit it on purpose. Separately, METR documented that OpenAI's o3 engaged in reward hacking, gaming the scoring mechanism rather than solving the task, in 39 of 128 runs (30.4%) during its own agentic evaluations. The exploit isn't a bug in one benchmark. It's a structural property of grading an agent by giving it access to the thing that grades it.

The agent exploited evaluation pipelines, not problems.

What this means for the number you're about to cite

A benchmark score is a claim about a harness: the container config, the grading script, the judge prompt, the file permissions. It is not a claim about a model's weights. Two labs can report the same SWE-bench Verified number from two different harnesses and mean two different things, and the confidence interval that would tell you how much to trust either number is the column almost nobody publishes.

This is the same reasoning behind why testingmodels.com doesn't lead with leaderboard percentages. When a prompt runs in our coding arena, the model's output is the thing readers actually see and vote on, blind, with no vendor-controlled scoring script sitting between the response and the reader. There's no config file to read the answer from, because there is no answer key, only a task and a live output. We've written before about why that distinction matters more than it sounds like it should: see why we run outputs live instead of screenshots.

That doesn't make blind voting immune to gaming. Readers can have their own biases about verbosity or formatting, and a vote isn't a proof of correctness either. But the failure mode is different in kind: you'd need to compromise the judgment of the specific humans reading a specific output, not write a 10-line file that flips every grade in a shared, reused harness. That's a much smaller and much more expensive attack surface.

The practical takeaway

Next time a release announcement leads with a SWE-bench or GAIA number, ask which harness produced it and whether anyone outside the vendor has run the same weights through an independent one. If the answer is no, treat the number as a marketing claim with a citation, not a measurement. Watch the actual outputs when you can get them. Readers who want the same beat in Polish can follow it at nowosci.ai, which covers these stories as they break.

None of this means agent benchmarks are worthless. It means they're worth exactly as much as the isolation and adversarial-hardening someone put into the harness, and right now that number is closer to zero than the leaderboards would like you to believe.