Every month brings a chart with a new model at the top, and every board that buys AI is eventually shown one. April 2025 was the month the charts came under scrutiny. Meta's Llama 4 arrived with a leaderboard result that turned out not to describe the model anyone could use, and a paper from a group of academic and industry researchers showed the practice behind it was widespread. This briefing sets out what happened, how the main leaderboards actually work, and how a business should read one.
How the Arena actually scores
Chatbot Arena, the most-watched of the boards, works by showing an anonymous pair of models the same prompt and asking the user to pick a winner; identities are revealed only after the vote. Those pairwise results are fed into a Bradley-Terry model, a statistical method from 1952 for estimating relative strength from wins and losses, and the coefficients are rescaled into an Elo-style number with bootstrap confidence intervals. The founding paper reported more than 240,000 votes by the time of publication.[1]
One refinement matters for reading any Arena score. Human raters reward polish, so in August 2024 the operators added style control: answer length, markdown headers, bold text and lists are entered into the same regression as confounders, so a model's published score reflects its strength net of those habits. The reranking moved real models, pushing some small models down and Claude and Llama variants up.[2] A score is therefore a measure of preference under those controls, from the population that votes, on the prompts they choose to type. It is a real measurement of a real thing. It is not a measure of whether a model can do your job.
What Meta released, and what ranked
On 5 April Meta published two open-weight models, Llama 4 Scout and Llama 4 Maverick, and previewed a third, Behemoth, still in training. Scout has 109 billion parameters with 17 billion active across 16 experts and claims a ten-million-token context; Maverick has 400 billion with 17 billion active across 128 experts; Behemoth was described at nearly two trillion parameters.[3] The models ship under Meta's own licence rather than an open-source one, with a cap for companies above 700 million monthly users and a restriction on multimodal use for companies domiciled in the EU.[4]
The launch materials pointed to a second-place Arena score of 1417. Within days it emerged that the model tested was an "experimental chat version" optimised for conversationality, which reviewers found notably fond of emoji and long answers, and not the weights Meta had released. When the public Maverick was submitted it ranked about thirty-second, below models many months older.[5][6] The Arena's operators said publicly that Meta should have made clear the tested model was customised, and changed the rules so the evaluated model must be the released one.[7] Meta's head of generative AI, Ahmad Al-Dahle, denied the sharpest accusation and offered an explanation: "We've also heard claims that we trained on test sets, that's simply not true", and the variable quality people were seeing was "due to needing to stabilize implementations".[8]
There was a second problem underneath the first. Scout's ten-million-token context was supported at launch only by needle-in-a-haystack retrieval tests, which show a model can find a planted string, not that it can reason across the document. Nathan Lambert, who writes on open models, noted the same week that this was "seen as a necessary condition, but not sufficient", and that a company launching a flagship with no more than that was doing something strange.[9] Independent long-context comprehension tests, which ask a model to synthesise facts scattered through a text rather than locate one, put Scout far below the leading closed models well inside its advertised window.
The Leaderboard Illusion
- 27
- Private Llama 4 variants Meta tested on the Arena in a single month before the public release [10]
- 39.6%
- Share of all Arena comparison data received by Google and OpenAI between them; 62.8% for the top four labs [10]
- 205
- Models the paper found silently removed from the comparison pool, against 47 marked deprecated [10]
On 29 April a paper titled The Leaderboard Illusion, led by Shivalika Singh with Sara Hooker and colleagues at Cohere Labs, Princeton, Stanford, MIT, Waterloo, the Allen Institute and the University of Washington, set out how the Arena had come to work in practice. Working from two million battles across 243 models and 42 providers, the authors found that some providers could test many private variants and publish only the best, which inflates a score without improving a model; Meta's 27 variants in March were the extreme case, and the figure excludes the vision and code boards. Proprietary models were sampled far more often than open ones, so Google and OpenAI between them received an estimated 39.6% of all comparison data and the top four labs 62.8%, while the least-favoured provider saw about a tenth of the maximum daily sampling rate. They estimated 205 models had been silently dropped from the pool against 47 officially deprecated, the deprecations falling hardest on open models.[10]
Two of their measurements matter most to a reader of charts. Testing ten private variants was worth roughly a hundred points on a provider's best score. And a model trained on Arena-style data improved sharply on Arena-style evaluation, with win rates more than doubling as the share of such data rose, which is the definition of teaching to the test. The authors' conclusion was that the incentives push toward "overfitting to Arena-specific dynamics rather than general model quality", and they proposed fixes: cap concurrent private variants, ban score retraction, retire models evenly across categories, sample actively, and publish the deprecation and sampling data.[10] The Arena's operators disputed parts of the analysis publicly, which is how this should work.
Why the static benchmarks stopped working
The Arena's problems are the mirror image of the fixed benchmarks'. A test with public answers ends up in training data, and its questions are not always right in the first place. Three measurements from the period make the point concrete. Scale AI's researchers built GSM1k, a fresh test matched in style and difficulty to the standard grade-school maths benchmark, and found some model families dropped by as much as 13 percentage points on the new questions, with the gap correlating with how readily a model reproduced the old ones.[11] A manual re-annotation of 3,000 questions from the most-cited knowledge benchmark found 9% were faulty, rising to 57% in one subject.[12] And an MIT group rebuilding fifteen popular benchmarks with the label errors removed found frontier models still failing simple problems once the ambiguity was stripped out.[13]
The response was harder tests with answers held back. Epoch AI's FrontierMath, launched in November 2024 with 300 unpublished research-level problems, saw no model exceed 2% at launch; in January 2025 it emerged that OpenAI had funded the benchmark and had access to most of the problems, which Epoch conceded it should have disclosed earlier, its associate director saying "we made a mistake in not being more transparent".[14] Humanity's Last Exam, published in January 2025, assembled 2,500 expert-written questions chosen because frontier models failed them.[15] ARC-AGI-2, launched on 24 March, was built so that ordinary people solve it and models do not: frontier models scored between 0% and 4% at launch against a human panel average of 60%.[16] LiveBench refreshes a share of its questions every month from material published after the models' training cut-offs.[17] Each of these buys time. Each will saturate in turn.
How to read one
- Check the model on the board is the model on sale. The Llama 4 episode was exactly this, and a chart from before April 2025 may not tell you.
- Ask what the score measures. The Arena is preference in a chat window under style controls; a coding benchmark is pass rate on fixed repository tasks; neither is your invoicing workflow.
- Prefer boards that hold answers back, refresh their questions and publish their method, and treat a score quoted by the vendor as the vendor's.
- Look at the spread, not the rank. When the top five sit within a few points, the ranking is noise and the price, the licence and the hosting are the decision.
- Run your own. Twenty representative tasks from your own work, scored by your own people, beat any public board for the question of whether a model will do your job. The practitioners who write about this say most failed AI products fail for want of an evaluation system rather than a better model.[18]