Research group Epoch AI published reviews of 21 popular benchmarks used to compare AI models and rated 10 of them as flawed. In SWE-bench Verified, which measures coding ability, it found faulty tests in 59.4% of the audited tasks, or at least 16.4% of the whole set. In Humanity's Last Exam, 22 of the 48 sampled questions had errors in their answers or wording. The group publishes findings per benchmark so they can be fixed or read with caution.

What it means: When an AI company launches a new model, benchmarks are usually the headline argument: a score on SWE-bench or Humanity's Last Exam becomes the title. Epoch's review shows that a large share of those numbers contains noise, since tests that reject correct solutions or questions with wrong answer keys can shift a ranking by several points. For Greek businesses choosing a model or AI tool for coding, customer service or document analysis, the conclusion is not to decide on a two or three point gap in public tables. It is more reliable to build a small test set of your own real tasks, for example 30 to 50 typical requests with a known correct answer, and measure accuracy, time and cost. The same applies to vendors selling AI solutions: claims should come with a demonstration on the customer's data. Greek is also a language that international benchmarks rarely cover, which makes in-house evaluation even more valuable. Epoch's work also helps anyone writing tender specifications to ask for more meaningful measurements than headline scores, and to require that results can be reproduced.

Read the original on Epoch AI

Related service: Custom software

Related stories