Hugging Face Built a Community Benchmark System. The Vendor Still Holds the Delete Button.

Hugging Face's Every Eval Ever (EEE) project launched in February 2026 with a genuinely good idea: a shared schema so an evaluation result carries its own paperwork, who ran it, what model it hit, how it was accessed, what the generation settings were, instead of showing up as a bare number in a tweet or a marketing deck. Paired with Community Evals, also launched that month, the two systems now feed scores directly onto a model's Hugging Face page. Read the fine print, though, and the feature that was supposed to open benchmarking up to anyone still leaves the door locked from the inside.

What actually shipped

The EvalEval Coalition, which built EEE, describes itself as the first cross-institutional effort to standardize how both first-party and third-party evaluators report results. In practice, a result on a model card now carries one of three badges: author-submitted (the model's own creator ran it), community-submitted (anyone filed a pull request with the right YAML), or independently verified (an official organization signed off, marked with a checkmark on EvalEval itself). The converter that turns raw eval logs into this format currently supports four benchmarks: MMLU-Pro, GPQA, Humanity's Last Exam, and GSM8K.

• EEE and Community Evals both launched February 2026, now interoperable
• Badge tiers: author-submitted, community-submitted, independently verified (checkmark)
• Converter covers four benchmarks so far: MMLU-Pro, GPQA, HLE, GSM8K
• Anyone can open a PR to attach a score to any model's page

The line that undoes it

Hugging Face's own post is straightforward about the one power that didn't move: "the model author can close PRs or hide results on their own repo." A stranger can run GPQA against a model and file the result. The badge will say community-submitted, which is honest labeling and better than nothing. But if that score is bad, the account that controls the model page can decline the pull request or hide it after the fact, and the page reverts to showing whatever the author already chose to post.

The model author can close PRs or hide results on their own repo.

That single sentence is the whole story. A system that lets anyone submit a score but lets the subject of the score veto which submissions survive isn't decentralizing evaluation, it's crowdsourcing the labor of testing while keeping the curation in the same hands it was always in. The badges make the results look more rigorous. The close-PR button means rigor is still optional for the party with the biggest incentive to skip it.

Four benchmarks that are already worn out

Even a fully honest submission through this pipeline runs into a second problem: GSM8K has been near ceiling for frontier models for two years and shows up in enough pretraining crawls that a clean score proves less than it used to. HLE was built specifically to be hard, which is good, but individual question sets have leaked into forums and repos since its release, and a model's HLE number tells you as much about what got scraped into its training data as what it can reason through cold. MMLU-Pro and GPQA are sturdier, but four benchmarks, however you badge the submissions, is a narrow slice of what a model needs to do well to be worth paying for.

Why we run it differently

None of this is a knock on the engineering, standardizing eval metadata is worthwhile work and better than the free-for-all it replaces. It's a reason to be specific about what a badge does and doesn't guarantee. On testingmodels.com, a model's output either survives a blind vote against the same prompt run through several other models, live, or it doesn't. There's no repo owner in that loop who can close a losing transcript. Readers see the raw output next to the competitors', unlabeled until after they've judged it, which is the same discipline behind why we run outputs live instead of screenshots: the thing you can't suppress after the fact is the thing worth trusting.

If you want the reasoning-effort and pricing side of this same argument, in Polish, nowosci.ai has been tracking the same string of announcements this week. Our own news feed has the day-by-day version in English.

The honest reading of Every Eval Ever is that it's a provenance layer bolted onto a system that was never short on provenance problems, it was short on a mechanism that can't be edited by the party being graded. Badges are metadata. A delete button is still a delete button.