🔍 Read the full analysis: How Researchers Can Use LLMs To Check An Idea’s Novelty on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models produced unstable judgments of scientific idea novelty across evaluation setups. In one pairwise test, small prompt changes shifted accuracy by more than 50 percentage points; retrieval and additional reasoning did not consistently help. The team released a benchmark, data, code and an evaluation pipeline, while warning that some novelty labels are imperfect.
Researchers at Ai2 report that six large language models gave unstable judgments of whether scientific ideas were novel, with small prompt changes shifting accuracy by more than 50 percentage points in one experiment, as detailed in the original analysis. The team has released a benchmark, data, code and an automated pipeline to help researchers test such systems, which may be used to screen ideas before costly literature reviews or experiments.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, as well as Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Researchers assessed the models using both absolute novelty ratings and pairwise comparisons. The authors report that judgments varied with choices such as prompt wording and evaluation format. In one pairwise experiment, changing a few words in the prompt moved accuracy by more than 50 percentage points.
The team also tested whether adding literature retrieval or giving models more reasoning would improve performance. Neither change helped consistently, according to the researchers. Two systems built specifically to judge novelty also performed worse than a simple prompted Claude Opus 4.6 baseline in the reported comparison.
For one benchmark set, the researchers selected 154 ideas from highly rated accepted ICLR 2026 papers whose reviewers praised originality without dispute, and 145 ideas from low-rated rejected papers where reviewers raised originality concerns without praise. An annotator agreed with the direction of the extracted novelty signal in 47 of 50 checked cases; the authors say the three disagreements did not change the review classifications.
Why Unstable Novelty Scores Matter
Research groups may use LLMs to sort candidate ideas before deciding which deserve expert review, a literature search or experimental work. If a model’s verdict shifts substantially after a minor prompt edit, its score may reflect evaluation choices as well as the idea itself. That makes it harder to compare results across teams or to treat a single high score as dependable evidence of originality.
The finding also bears on efforts to use AI systems to support scientific discovery. Idea generators and literature-synthesis tools need credible ways to assess whether a proposal adds something not already known. The study does not establish that all LLM novelty assessments fail; it shows that success in one chosen setup is not, by itself, evidence of reliable performance across setups.
By publishing its materials, Ai2 gives other researchers a way to compare models and evaluation decisions against shared test cases. That could help identify which parts of the process cause judgments to vary, though the release does not establish that a more stable method has already been found.
Products Worth Considering
Logitech MX Keys S Wireless Keyboard, Graphite
Apple iPad Air (4th Generation, 10.9-inch, Wi-Fi, 256GB) – Space Gray
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Novelty Tests
A novelty judge is a model asked to assess whether an idea is new relative to earlier work. It can rate an idea on its own or compare two proposals. The study authors say there is no standard recipe for building these evaluations: researchers can select different models, prompts, reasoning budgets, comparison formats and literature-retrieval tools. They also argue that judges are often assessed on human-written papers even when the intended use is screening AI-generated ideas.
Ai2’s benchmark includes a separate setting that pairs the 154 human-authored ideas with 154 ideas produced by a basic language-model generator. The researchers treated the generated proposals as less novel, but describe those labels as weak and not individually verified. In an expert check of 30 pairs, selected heavily from cases the strongest judge had got wrong, the expert preferred the human idea in every pair and judged none of the generated ideas novel. That check provides evidence about those pairs, not the full set.
The authors caution that acceptance or rejection alone is not a sound measure of novelty: papers may succeed or fail for reasons unrelated to originality. Their benchmark instead uses reviewer comments as signals, which are still not a definitive measure of originality.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark’s labels rely partly on reviewer comments and selection criteria, rather than a definitive test of whether an idea is original. The generated-idea set has a further limitation: the researchers assumed those ideas were less novel than the selected human ideas, while acknowledging that they did not verify every generated proposal individually. The expert assessment covered 30 pairs, not all 154 generated ideas.
The supplied study description does not give full results for every model, prompt, reasoning budget or evaluation format, or state the release’s calendar date. It also does not show whether the findings apply to other academic fields, different idea generators or systems with broader access to research literature. The authors say some conclusions about disagreement between setups do not depend on benchmark labels, but the material provided does not detail how broadly that result holds.
Accordingly, the study supports a concern about reliability across evaluation choices; it does not settle which design, if any, can produce dependable novelty judgments in routine research use.
Further Tests Across Research Fields
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help researchers build additional evaluation sets. The next step is for other teams to test whether the reported instability appears across disciplines, idea sources and judge designs, and to examine methods that could make results more consistent.
The release description does not announce a specific follow-up study or timeline. Until broader tests are available, researchers considering LLM novelty scores will need to treat them as setup-dependent assessments, rather than conclusive evidence that an idea is new.
Key Questions
What did the Ai2 researchers find?
They report that six LLMs gave unstable scientific-idea novelty judgments across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, using absolute ratings and pairwise comparisons.
Did retrieval or more reasoning make the judgments reliable?
Not consistently, according to the authors. They report that literature retrieval and additional reasoning did not reliably improve novelty judgments across the tested setups.
How reliable are the benchmark labels?
They are useful signals, not definitive novelty ground truth. One set uses reviewer comments about originality; the generated-idea set treats proposals as less novel without individually verifying every label. An expert check covered 30 pairs, not the whole benchmark.
What has Ai2 released?
Ai2 released the study, benchmark data, code and an automated pipeline for building additional evaluation sets. The source material does not name a specific follow-up study or schedule.
Primary source: Hugging Face · via ThorstenMeyerAI.com
