🔍 Read the full analysis: UK AISI, EvalEval, And The Push For Reproducible AI Benchmarks on ThorstenMeyerAI.com
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
The UK AI Security Institute is publishing selected results from five benchmarks across six frontier models through EvalEval’s Evaluation Cards, alongside verification, context and configuration details. The release accompanies a paper examining how inference-time compute and evaluation protocols affect measured performance; its coverage does not include every AISI evaluation or transcript.
The UK AI Security Institute (AISI) is publishing selected results from frontier AI evaluations through EvalEval’s Evaluation Cards, with verification, evaluation context and configuration details intended to help readers interpret how the scores were produced, as detailed in the original analysis. The release accompanies AISI’s paper on inference-time compute and evaluation protocols and covers five benchmarks across six models, as well as two cyber evaluations using a separate, partly overlapping model set.
The five benchmarks in the paper’s main experiment are HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro and Terminal-Bench 2.0. The listed models are Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4. AISI has also shared results for Cyber CTFs and The Last Ones. The source material does not list the models used in those cyber evaluations, and says their model set overlaps only partly with the main experiment.
The records are associated with AISI’s paper, How Inference Compute Shapes Frontier LLM Evaluation. That paper examines how results depend on the amount of inference-time compute and on evaluation protocols. In its analysis of Humanity’s Last Exam, AISI tracks the cumulative share of attempted tasks solved within a given token count, counting each task’s earliest observed success. In runs where models received correctness feedback from an oracle after each attempt, they solved additional tasks as token use increased.
EvalEval says the cards combine verified results, evaluation context and configuration information. Its platform organizes benchmark metadata, evaluation-run data and model metadata in a shared format. AISI’s announcement says publicly reported methods and findings are being made available where appropriate; it does not say that every evaluation or underlying transcript is included.
Why Evaluation Conditions Matter
Benchmark scores are often compared across models as though they were produced under the same conditions. AISI’s analysis shows why that assumption can be unreliable: measured performance on Humanity’s Last Exam varied with inference compute and with whether a model received correctness feedback between attempts. A score separated from those details can leave readers uncertain about what it measures.
Publishing a result alongside its setup can help researchers and practitioners inspect individual runs and identify when similar-looking scores came from different conditions. This is relevant to AI research, model development and policy work that uses evaluations as evidence about model capabilities. The cards provide reporting infrastructure; they do not, by themselves, establish which benchmark or protocol is best, or prove that another researcher can reproduce every result.
Top picks for "aisi evaleval push"
As an affiliate, we earn on qualifying purchases.
The release builds on earlier work between AISI and EvalEval that began at a joint workshop alongside NeurIPS 2025. EvalEval says feedback from AISI helped shape Every Eval Ever (EEE), a shared schema for documenting evaluations. The current cards apply that infrastructure to selected AISI methods and findings that have been publicly reported.
AISI has also worked on evaluation efficiency through OptStop, statistical rigor through HiBayES, and standardisation in areas including transcript analysis and capability elicitation. EvalEval’s Evaluation Cards bring results together with benchmark and model information. These efforts address a reporting problem: evaluation results can appear in formats that omit details needed to interpret a run, while repeating expensive evaluations may not be practical.
“AISI is using EvalEval’s infrastructure to openly share evaluation results, supporting more reproducible and verifiable evaluation science.”
— EvalEval Coalition
What the Cards Do Not Yet Show
The announcement does not specify the total number of records or transcripts, which setup fields are available for every benchmark, or whether outside researchers have independently reproduced the results. It also does not enumerate the model set for Cyber CTFs and The Last Ones. The release is described as covering publicly reported material where appropriate, so its scope should not be treated as a complete archive of AISI’s evaluation work.
No record-by-record publication dates or process for resolving disagreements between results produced under different protocols are given. Those details would help readers judge coverage and compare records consistently. The available information also does not establish how complete or uniform future contributions to the shared schema will be.
Wider Use of Evaluation Cards
EvalEval says it expects to continue standardising and sharing evaluations with AISI and other evaluation organisations. Its proposed next step is broader use of Every Eval Ever: model developers can submit verified results, and evaluation developers can report benchmarks and run data using the schema. Researchers in evaluation, governance and policy can explore Evaluation Cards by benchmark or model.
No further release date or adoption milestone was specified. Wider use could make cross-study comparisons easier, but how useful those comparisons are will depend on contributors publishing records with consistent and sufficiently complete information.
Key Questions
What is AISI releasing?
AISI is sharing selected evaluation results through EvalEval’s Evaluation Cards, which pair results with verification, evaluation context and configuration information.
Which benchmarks and models are covered in the main experiment?
The five benchmarks are HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro and Terminal-Bench 2.0. The six listed models are Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4.
Are the cyber evaluations using the same models?
Not necessarily. Cyber CTFs and The Last Ones use a separate, partly overlapping model set. The announcement does not enumerate that set.
Does this release include every AISI result and transcript?
No such claim is made. The announcement says publicly reported methods and findings are being shared where appropriate, but does not describe the release as a complete archive.
Why include evaluation setup details with scores?
AISI’s paper reports that results can depend on inference-time compute and protocol choices, including whether a model receives correctness feedback between attempts. Recording those conditions can help readers interpret and compare scores.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
