🔍 Read the full analysis: The Referee Shortage: AI Made Doing Cheap And Checking Expensive on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A source article describes a widening gap between AI’s ability to produce work and people’s capacity to verify it, using examples from mathematics, software and contract work. The figures point to a possible review bottleneck, but several software metrics come from vendors, and the long-term effects on job training and accountability remain uncertain.
A report published this week argues that AI is making work faster to produce without reducing the human effort needed to verify it, a mismatch that could constrain how much machine-generated research, code and professional work organisations can safely use. Its examples include 722 mathematics manuscripts attributed to an OpenAI programme and software-industry figures that suggest reviews are taking longer or being skipped.
The report says OpenAI posed about 4,000 mathematical problems and produced 722 manuscripts across 372 topic families, with an average result taking about three hours of compute. Some results were formally checked using Lean, a proof assistant. OpenAI cautioned that some unformalized results could have issues. The report contrasts that volume with the extensive expert attention given to a counterexample to an Erdős conjecture from the same programme, but does not provide a full account of the verification process or its timeline.
In software, the report cites data from companies that track engineering teams. Faros AI found teams merged 98% more pull requests during high-AI-adoption periods while review time rose 91%. LinearB’s analysis of 8.1 million pull requests across 4,800 organisations found AI-generated changes waited 4.6 times longer for review to begin and were accepted 32.7% of the time, compared with 84.4% for human-written changes. A separate peer-reviewed 2026 study reportedly found 61% of AI-agent pull requests received no human review before being merged or closed.
The report also describes OpenAI’s partnership with contract-software company Ironclad, saying the model GPT-6 Astra averaged 55% of evaluation criteria across 11 contracting tasks. That result indicates performance against the stated evaluation, not that the remaining criteria represent a 45% error rate in real contracts. The report does not supply the evaluation methodology or task-level results.
The referee shortage: AI made doing cheap and checking expensive
OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.
Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.
Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.
Real progress. Someone still has to find the other 45% before the work can be used.
Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.
A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.
Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.
of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).
of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.
OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.
Budget review hours next to model spend.
Provers, types, tests, policy engines.
Experts only where consequences are high.
Who profits from generation pays for checking.
Keep some production human for learners.
The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.
Review Capacity Shapes AI Adoption
If AI raises the volume of work faster than organisations can check it, review becomes a practical limit on deployment. More generated code or research does not automatically mean more usable output: someone still needs to judge whether it is correct, relevant and safe to rely on. The report’s central argument is that this verification work may become more valuable even as the production work becomes cheaper.
That could affect staffing and workflow design. Organisations may face a choice between slowing output for review, accepting greater risk, or directing scarce experts toward the most consequential work. The evidence cited does not establish that every field is experiencing the same pattern, but it highlights a problem for any workplace where approval, professional responsibility or safety checks are required.
Products Worth Considering

Jackery Explorer 300 Portable Power Station, 292Wh

Bose Smart Dolby Atmos Soundbar with Alexa, A.I. Dialogue Mode, and Bluetooth, Black

Sony WH-1000XM6 Wireless Noise-Cancelling Headphones (Black)
As an affiliate, we earn on qualifying purchases.
Three Fields, One Reported Pattern
The report connects developments in mathematics, software and contract work through a distinction between generating an answer and deciding whether to trust it. Automated proof checking can establish whether a proof follows from a stated theorem, for example, but human experts may still need to assess whether the theorem is the right one and whether the result matters. Similarly, software tests check specified behaviour, not whether the tests cover the user’s actual needs.
The report says some software findings come from firms selling code-review or engineering tools, a commercial interest readers should account for when interpreting the figures. Its cited measures also use different samples and definitions: adoption periods, review-start delays, acceptance rates and human-review rates are not interchangeable measures. Taken together, they suggest pressure on review workflows, but they do not by themselves prove a single cause or quantify the overall effect across the industry.
The report’s further concern is the training pipeline. Junior workers often develop judgment by doing the underlying work before reviewing others’ work. If AI substantially replaces those entry-level tasks, the pool of experienced reviewers could eventually shrink. That is a forecast, not an outcome demonstrated by the figures presented.
Evidence Leaves Key Questions Open
The source material does not provide links, full methods or publication details for every cited dataset, so the figures cannot be independently assessed here. In particular, the software results come from different samples and may define review, adoption and AI-generated work differently. The report itself says several data providers sell code-review tools, making independent replication important.
It is also unclear how much of the measured change is caused by AI rather than other changes in team size, workflow or project mix. The report offers no comparable long-term data showing whether review quality, defect rates or delivery outcomes have worsened. Its examples support a concern about review capacity, but they do not establish that AI has made verification more expensive in every setting.
The contract-evaluation account lacks enough detail to determine what the 55% score means in practical legal work. The report also does not explain how the 722 mathematical manuscripts were selected for publication from the larger set of problems, or provide a systematic measure of how much expert time was required to validate them.
Measure Review Alongside Output
The next useful evidence would track output and verification together: how much review time AI-assisted work requires, how often defects are found, what happens after deployment, and who accepts responsibility for approval. Independent studies using transparent definitions would help establish whether the reported software patterns hold beyond vendor datasets.
For employers, the question is not only how much work AI can generate, but how to maintain expert oversight and train future reviewers. The source report argues that organisations should protect opportunities for junior staff to learn the work they may later need to judge. Whether that approach becomes common, and whether review roles expand or change, remains to be seen.
Key Questions
What is the main development described?
A report argues that AI-generated work is growing faster than human review capacity, citing examples from mathematics, software and contract work. It presents this as an emerging bottleneck rather than a settled finding across all industries.
Did OpenAI publish 722 verified mathematics results?
The source report says the programme produced 722 manuscripts from about 4,000 problems. It says some results were formally checked in Lean and quotes OpenAI warning that some unformalized results could have issues. The material does not say all 722 were independently verified.
What do the software figures show?
The cited figures point to more pull requests, longer waits for review to begin, lower acceptance rates for AI-generated changes in one dataset, and many AI-agent pull requests receiving no human review in a separate study. The measures come from different sources and should not be treated as one combined estimate.
Does the report prove AI makes review more expensive?
No. It argues that verification may become a bottleneck, but the cited figures do not establish a general causal effect or show that review costs have risen in every field. More independent, comparable evidence is needed.
Why could junior job training be affected?
The report warns that entry-level work can help people build the expertise needed to review and approve work later. If AI replaces too much of that practice, organisations could face a smaller future pool of experienced reviewers; this remains a forecast, not a demonstrated outcome.
Source: ThorstenMeyerAI.com
