MMLU has wrong answers, leaked questions, and 14-point contamination gaps. Why AI benchmarks broke — and what would fix them.
In June 2024, researchers at the University of Edinburgh checked the exam's homework: working through 3,000 questions from MMLU — the Massive Multitask Language Understanding benchmark, the most-cited test in AI — they found 6.5 percent had wrong answer keys, ambiguous wording, or unparseable options. In the virology subset, 57 percent were defective; a third had completely wrong labeled answers. The team, led by Aryo Gema, published a corrected MMLU-Redux and gave it away free. Two years later, no flagship launch has switched to it. Every press release still quotes the broken original — to one decimal place.
When the answer key is wrong for one question in fifteen, a model improving from 88.1 to 88.7 percent hasn't necessarily learned chemistry — it may have learned the key's mistakes. That is Goodhart's law at industrial scale: millions spent optimizing against a target that is measurably defective at exactly the advertised precision.
Work in late 2024 found models have systematic biases toward particular answer positions: reorder the options and a score can swing by as much as 24 percentage points. The benchmark wasn't measuring knowledge. It was measuring an interaction between knowledge and layout.
Benchmarks must be public to be useful; public means scrapeable; scrapeable means test questions end up in the training data of the next generation. Microsoft built MMLU-CF, a contamination-free variant rephrasing questions and shuffling options. The results were stark. GPT-4o fell from 88.0 percent to 73.4. GPT-4-Turbo dropped from 86.5 to 70.4. Meta's Llama 3.3 70B lost 17.5 points, from 86.3 to 68.8; Microsoft's own Phi-4 fell 17 points, from 84.8 to 67.8. Every model shed 14 to 17.5 points the moment the answers couldn't be recalled.
When researchers asked GPT-4 to guess the missing answer options in MMLU questions — impossible without having seen the test — it guessed correctly 57 percent of the time. When studies split benchmark questions by release date, models consistently score 8 to 15 percentage points better on questions published before their training cutoff than after. Dekoninck and colleagues showed in 2024 the leak needn't even be the test itself: fine-tuning on benchmark-style samples lifted scores 8 to 11 percent on held-out questions. Aleph Alpha's 2025 experiments found detectors caught only 13 percent of leaked TriviaQA questions — the industry's decontamination filters are, at best, sieves.
By 2025, the top five frontier models compressed into a band above 89 percent on MMLU, separated by less than two points — smaller than the noise from prompt formatting alone. HumanEval, OpenAI's 164-problem coding benchmark, shows the same arc: top models clear 90 percent on tasks — write a function that reverses a string — bearing little resemblance to real software engineering. Sebastian Raschka, one of the field's most-cited educators, concluded that benchmark numbers are "no longer trustworthy indicators of LLM performance."
The gaming has a name: benchmaxxing. The techniques include checkpoint selection (evaluate many runs, publish the best), format overfitting (fine-tune on multiple-choice style), and eval-specific prompting (tune system prompts to the scoring harness). The Allen Institute for AI's Fluid Benchmarking delivered the quietest indictment in 2025: 100 well-chosen questions rank models more accurately than the full 14,000. The other 13,900 were too easy, too hard, or actively misleading.
Beneath this sits a category error. Most famous benchmarks measure what a model learned in training. Almost nobody measures, with equal rigor, inference performance — the thing every buyer actually pays for. An MMLU score says nothing about time-to-first-token at production load, throughput in tokens per second, latency degradation at long context, or cost per million tokens on real traffic. Training benchmarks like MLCommons' MLPerf measure training throughput; they were never designed to answer what a model costs per query or how fast users see the first word.
That gap is why independent inference benchmarking emerged as a discipline. Artificial Analysis, founded by George Cameron, publishes its configurations and runs separate inference-performance leaderboards — and even it demonstrates the versioning problem: its v4.0 methodology removed MMLU-Pro and LiveCodeBench, rendering older scores incomparable with newer ones. Stanford's HELM, once the academic gold standard for transparent evaluation, entered maintenance mode on June 1, 2026. The referees keep retiring or resetting; the scoreboard keeps printing decimals.
The honest defense deserves a hearing. MMLU still works as a sanity check: a model scoring below 80 percent almost certainly has real capability gaps. LiveBench and LiveCodeBench resist contamination by drawing questions from recently released sources; private held-out evaluations — Scale AI's SEAL program is the prominent example — test on sets that never touch the public internet. Nor is every score inflation: the underlying capability gains of the past three years are visible in production revenue, not just leaderboards.
But the defense concedes the essential point: benchmarks survive as sanity checks and rough filters, not as rankings — the decimal-point theater of launch-day press releases and procurement decisions made on two-point differences smaller than the test's own error rate.
The fix is not a harder test; harder tests contaminate on schedule. The fix is institutional: independent inference benchmarking as an industry utility, on the model of financial-market data rather than vendor marketing. The rules are simple to state: publish the prompts, seeds, harness, and configuration — every number reproducible by a third party. Measure inference, not just intelligence: time-to-first-token, tokens per second, p95 latency, price per million tokens, all on standardized workloads. Refresh items continuously from post-cutoff sources; report contamination methodology alongside scores; version ruthlessly and mark old scores incomparable when the ruler changes.
None of this requires new science. It requires a referee whose incentives point at accuracy rather than launch day — and the willingness of labs to submit to measurement they don't control. That willingness is the scarce resource. Until it arrives, every buyer faces the same arithmetic: published research puts score inflation between 6 and 17 points, the rankings at the top are noise, and the only measurements worth paying for are run on your own workload, at your own traffic pattern, by someone with no model to sell.
The category-defining name in AI inference intelligence — inferencebenchmarks.com — is currently available for acquisition by the firm positioned to own this space.
