Every time a company launches a new AI model, it publishes a table of scores: 87.4% on this test, 72.1% on that one. Those numbers come from evals, short for evaluations, which are structured tests that try to measure what a model can do, and what it might do that nobody wants.
Testing a model sounds simple. You ask it questions and mark the answers. In practice it is one of the hardest problems in AI, because models can be good at exams in ways that do not carry over to real work, and because the risks people worry about most are the hardest ones to test for.
The simple version
Testing an AI is like testing a new driver: the written exam is not the road test.
- A benchmark is a fixed exam. Every model gets the same questions, marked the same way.
- Scores let you compare models, but only on what that exam covers.
- If a model has seen the questions before, its score means much less.
- Red teams play the villain. They try to make the model misbehave on purpose.
- Outside testers check the homework, because companies usually mark their own.
This is the plain-English guide. For a deeper look at saturation, contamination and why scores often cannot be compared between models, read AI Benchmarks Explained: What the Scores Actually Measure.
Why it matters now
Three stories this week all came back to testing. OpenAI reported that its newest model passed 72.1% of COBOL tests on MultiLangBench, a benchmark it built itself, up from 10.9% for an earlier model, and said it would release the benchmark on GitHub so others can check. The EU’s AI Office sent its first formal requests for information to makers of the most powerful general-purpose models, covering security, external evaluation and how the models are used. And Anthropic gave six staff from METR, an independent evaluation group, internal access and the right to publish their risk assessments without a company veto.
The question underneath all three is the same: who does the testing, how, and how far can you trust the result?
How it works, step by step
Step 1: Benchmarks, the standard exam
A benchmark is a fixed set of tasks plus a marking scheme. Because every model sits the same paper, scores can be compared, like exam results across a school year.
Two well-known examples show the range. MMLU, published in 2020, is a multiple-choice test spanning 57 subjects, from elementary maths to law. SWE-bench, published in 2023, is closer to real work: it contains 2,294 problems taken from genuine GitHub issues in 12 popular Python projects, and a model only scores if the code change it writes makes the project’s own tests pass.
Step 2: Marking the answers
Multiple choice is easy to mark. Code can be marked by running it. Open-ended writing is harder: teams use human raters, or ask another model to act as a judge, and each method has its own blind spots.
Settings matter too. The same model can score differently depending on the exact prompt, how many attempts it is allowed and how much “thinking” time it gets. OpenAI’s headline COBOL figure, for instance, was measured at maximum reasoning effort. A score is a result under particular conditions, not a fixed property of the model.
Step 3: Leaks and ceilings
Benchmarks are usually published online, and models are trained on huge amounts of online text. If test questions end up in the training data, that is called data contamination. It is like a student who has seen the answer sheet: a high mark no longer proves much.
Researchers can check for this by writing fresh questions. In 2024 a team built GSM1k, a new set of grade-school maths problems designed to match a popular existing test, and found that some models scored noticeably worse on the fresh questions, a sign they had partly memorised the old ones.
The opposite problem is saturation. When the best models all score near 100%, a test stops telling them apart, so harder tests get built.
Step 4: Red teams and dangerous-capability tests
A benchmark asks “how good is it?” A red team asks “how could this go wrong?” Red teamers, people and increasingly automated tools, deliberately try to make a model reveal private data, help with a cyberattack or ignore its instructions.
Other evals put the model to work on long tasks with real tools, as an AI agent. METR measures a model’s task horizon: the length of task, timed by how long it takes skilled people, that the model completes successfully half the time. Its 2025 paper reported that this horizon had doubled roughly every seven months since 2019.
In Europe this kind of testing is now a legal duty for some. Article 55 of the EU AI Act requires providers of general-purpose models with “systemic risk” to evaluate them, including adversarial testing, and to document it.
Step 5: Who holds the pen
Think of a restaurant. It can check its own kitchen, it can hire an outside auditor, or a hygiene inspector can turn up. AI testing has all three. Companies test their own models first. Independent groups such as METR test with the company’s cooperation. Government bodies build tools and ask questions: the UK AI Security Institute publishes Inspect, an open-source framework for running evals, and the EU AI Office can now demand information.
A worked example: a council picks a chatbot
Imagine a local council wants a chatbot to answer residents’ questions about bin collections. Rather than trust a leaderboard, it builds its own eval.
- Collect real questions. Staff pull 200 questions from last year’s emails and write down the correct answer to each.
- Run every candidate the same way. Same prompt, same settings, several runs each, because answers vary from run to run.
- Mark. Facts such as the collection day are checked automatically. Two staff members score tone and clarity.
- Red-team it. Staff add 30 awkward inputs, such as a request for a neighbour’s address, or a message with a hidden line saying “ignore your rules”.
- Decide. Model A has the better public benchmark score. Model B gets more bin questions right and refuses the address request. The council picks B.
The lesson: the most useful test is the one that looks most like your actual job.
What people get wrong
“A higher score means a better model.” It means a higher score on that test, under those settings. A model can top a maths leaderboard and still be a poor fit for customer service.
“The launch numbers are independent.” Most launch figures are measured by the company itself. That is why AIooze briefs say when results are the company’s own, and why releasing a benchmark publicly, as OpenAI says it will with MultiLangBench, matters: it lets others try to reproduce the result.
“Passing safety tests proves a model is safe.” A test can show that a problem exists. It cannot prove that no problem exists, because nobody can test every possible input.
“Benchmarks are useless.” They are imperfect, but without a shared yardstick there would be nothing to compare at all. The fix is better and fresher tests, not no tests.
Where it is heading
- Longer, more realistic tasks. Tests are moving from single questions to multi-step work with real tools, as SWE-bench and METR’s time-horizon measure show.
- Regulators asking for evidence. The EU AI Office’s requests are a first formal step under the AI Act, not a finding or a fine, but they make evaluation records something a company may have to hand over.
- Outside testers with deeper access. Anthropic published a three-tier framework for outside evaluators, with METR at the top tier, and said every frontier AI company should adopt this kind of arrangement. No other company has committed to it yet.
- Fresh test sets. Approaches like GSM1k, writing new questions that have never been online, are one answer to contamination.
Key terms
- Benchmark
- Eval (evaluation)
- Red teaming
- Data contamination
- Task horizon
- AI agent
- Reasoning model
- Hallucination
Sources
- Hendrycks et al., “Measuring Massive Multitask Language Understanding” (MMLU), arXiv:2009.03300, 2020. arxiv.org/abs/2009.03300
- Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, arXiv:2310.06770, 2023. arxiv.org/abs/2310.06770
- Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” (GSM1k), arXiv:2405.00332, 2024. arxiv.org/abs/2405.00332
- Kwa et al. (METR), “Measuring AI Ability to Complete Long Software Tasks”, arXiv:2503.14499, March 2025. arxiv.org/abs/2503.14499
- Regulation (EU) 2024/1689 (AI Act), Article 55. eur-lex.europa.eu
- UK AI Security Institute, Inspect. inspect.aisi.org.uk
- OpenAI, low-resource programming languages post, October 2026. openai.com
- European Commission, speech by Executive Vice-President Henna Virkkunen, 14 October 2026 (SPEECH/26/2150). ec.europa.eu
- Anthropic, “An update on independent evaluators”, 15 October 2026. anthropic.com