The Referee Shortage: AI Made Doing Cheap And Checking Expensive
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: The Referee Shortage: AI Made Doing Cheap And Checking Expensive on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

A source article describes a widening gap between AI’s ability to produce work and people’s capacity to verify it, using examples from mathematics, software and contract work. The figures point to a possible review bottleneck, but several software metrics come from vendors, and the long-term effects on job training and accountability remain uncertain.

A report published this week argues that AI is making work faster to produce without reducing the human effort needed to verify it, a mismatch that could constrain how much machine-generated research, code and professional work organisations can safely use. Its examples include 722 mathematics manuscripts attributed to an OpenAI programme and software-industry figures that suggest reviews are taking longer or being skipped.

The report says OpenAI posed about 4,000 mathematical problems and produced 722 manuscripts across 372 topic families, with an average result taking about three hours of compute. Some results were formally checked using Lean, a proof assistant. OpenAI cautioned that some unformalized results could have issues. The report contrasts that volume with the extensive expert attention given to a counterexample to an Erdős conjecture from the same programme, but does not provide a full account of the verification process or its timeline.

In software, the report cites data from companies that track engineering teams. Faros AI found teams merged 98% more pull requests during high-AI-adoption periods while review time rose 91%. LinearB’s analysis of 8.1 million pull requests across 4,800 organisations found AI-generated changes waited 4.6 times longer for review to begin and were accepted 32.7% of the time, compared with 84.4% for human-written changes. A separate peer-reviewed 2026 study reportedly found 61% of AI-agent pull requests received no human review before being merged or closed.

The report also describes OpenAI’s partnership with contract-software company Ironclad, saying the model GPT-6 Astra averaged 55% of evaluation criteria across 11 contracting tasks. That result indicates performance against the stated evaluation, not that the remaining criteria represent a 45% error rate in real contracts. The report does not supply the evaluation methodology or task-level results.

At a glance
analysisWhen: Report published this week; underlying…
The developmentA report argues that rapid AI-generated output is increasing demand for human verification faster than review capacity is growing.
The Referee Shortage — Post-Labor
AI Dispatch · Post-Labor · 7 October 2026

The referee shortage: AI made doing cheap and checking expensive

OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.

One pattern, three fields
Mathematics
722
manuscripts, ~3h compute each

Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.

Software
+98% / +91%
more PRs merged / longer review

Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.

Professional workflows
55%
of criteria met — Astra on Ironclad

Real progress. Someone still has to find the other 45% before the work can be used.

Generation collapsed. Verification didn’t. (conceptual, not to scale)
Cost to produce a resultdown
Cost to check a resultnot down
No author intent

Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.

Checks the answer, not the question

A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.

Someone must be accountable

Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.

Illustrative: $1 of model time + 4 minutes of review at $45/hour. Halving the model price saves 12.5%; one extra review minute erases it. In that example, review is three-quarters of the bill.
What happens when referees run out — already visible
Rubber-stamping
61%

of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).

Triage by suspicion
38%

of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.

Producer as filter
~4,000 → 372

OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.

The apprenticeship paradox: reviewers are made by doing the work. The work AI absorbs — writing code, drafting contracts, proving lemmas — is exactly what trained the reviewers. Demand for judgement rises as its supply line shrinks.
What to do
Price verification

Budget review hours next to model spend.

Formalise checks

Provers, types, tests, policy engines.

Tier the review

Experts only where consequences are high.

Fund the referees

Who profits from generation pays for checking.

Protect apprenticeship

Keep some production human for learners.

The take

The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.

Sources: OpenAI maths release & Erdős verification as covered here; arXiv:2608.28997; OpenAI × Ironclad (6 Oct 2026); Faros AI; LinearB 2026 (8.1M PRs); Duma et al., EASE 2026 — via secondary reporting. Several code-review sources sell review tools. Review-cost example illustrative. Analysis is the author’s.
thorstenmeyerai.com

Review Capacity Shapes AI Adoption

If AI raises the volume of work faster than organisations can check it, review becomes a practical limit on deployment. More generated code or research does not automatically mean more usable output: someone still needs to judge whether it is correct, relevant and safe to rely on. The report’s central argument is that this verification work may become more valuable even as the production work becomes cheaper.

That could affect staffing and workflow design. Organisations may face a choice between slowing output for review, accepting greater risk, or directing scarce experts toward the most consequential work. The evidence cited does not establish that every field is experiencing the same pattern, but it highlights a problem for any workplace where approval, professional responsibility or safety checks are required.

Products Worth Considering

Bose Smart Dolby Atmos Soundbar with Alexa, A.I. Dialogue Mode, and Bluetooth, Black

Bose Smart Dolby Atmos Soundbar with Alexa, A.I. Dialogue Mode, and Bluetooth, Black

As an affiliate, we earn on qualifying purchases.

Three Fields, One Reported Pattern

The report connects developments in mathematics, software and contract work through a distinction between generating an answer and deciding whether to trust it. Automated proof checking can establish whether a proof follows from a stated theorem, for example, but human experts may still need to assess whether the theorem is the right one and whether the result matters. Similarly, software tests check specified behaviour, not whether the tests cover the user’s actual needs.

The report says some software findings come from firms selling code-review or engineering tools, a commercial interest readers should account for when interpreting the figures. Its cited measures also use different samples and definitions: adoption periods, review-start delays, acceptance rates and human-review rates are not interchangeable measures. Taken together, they suggest pressure on review workflows, but they do not by themselves prove a single cause or quantify the overall effect across the industry.

The report’s further concern is the training pipeline. Junior workers often develop judgment by doing the underlying work before reviewing others’ work. If AI substantially replaces those entry-level tasks, the pool of experienced reviewers could eventually shrink. That is a forecast, not an outcome demonstrated by the figures presented.

Evidence Leaves Key Questions Open

The source material does not provide links, full methods or publication details for every cited dataset, so the figures cannot be independently assessed here. In particular, the software results come from different samples and may define review, adoption and AI-generated work differently. The report itself says several data providers sell code-review tools, making independent replication important.

It is also unclear how much of the measured change is caused by AI rather than other changes in team size, workflow or project mix. The report offers no comparable long-term data showing whether review quality, defect rates or delivery outcomes have worsened. Its examples support a concern about review capacity, but they do not establish that AI has made verification more expensive in every setting.

The contract-evaluation account lacks enough detail to determine what the 55% score means in practical legal work. The report also does not explain how the 722 mathematical manuscripts were selected for publication from the larger set of problems, or provide a systematic measure of how much expert time was required to validate them.

Measure Review Alongside Output

The next useful evidence would track output and verification together: how much review time AI-assisted work requires, how often defects are found, what happens after deployment, and who accepts responsibility for approval. Independent studies using transparent definitions would help establish whether the reported software patterns hold beyond vendor datasets.

For employers, the question is not only how much work AI can generate, but how to maintain expert oversight and train future reviewers. The source report argues that organisations should protect opportunities for junior staff to learn the work they may later need to judge. Whether that approach becomes common, and whether review roles expand or change, remains to be seen.

Key Questions

What is the main development described?

A report argues that AI-generated work is growing faster than human review capacity, citing examples from mathematics, software and contract work. It presents this as an emerging bottleneck rather than a settled finding across all industries.

Did OpenAI publish 722 verified mathematics results?

The source report says the programme produced 722 manuscripts from about 4,000 problems. It says some results were formally checked in Lean and quotes OpenAI warning that some unformalized results could have issues. The material does not say all 722 were independently verified.

What do the software figures show?

The cited figures point to more pull requests, longer waits for review to begin, lower acceptance rates for AI-generated changes in one dataset, and many AI-agent pull requests receiving no human review in a separate study. The measures come from different sources and should not be treated as one combined estimate.

Does the report prove AI makes review more expensive?

No. It argues that verification may become a bottleneck, but the cited figures do not establish a general causal effect or show that review costs have risen in every field. More independent, comparable evidence is needed.

Why could junior job training be affected?

The report warns that entry-level work can help people build the expertise needed to review and approve work later. If AI replaces too much of that practice, organisations could face a smaller future pool of experienced reviewers; this remains a forecast, not a demonstrated outcome.

Source: ThorstenMeyerAI.com

You May Also Like

Forezai · Polybot: When the AI Disagrees With the Odds

Polybot, an AI trading experiment, attempts to identify when an AI’s probability estimates diverge from market prices, raising questions about market prediction and risk.

The SSD Squeeze: Why Storage Joined the Party

Record-breaking NAND price increases driven by AI storage needs and factory competition are reshaping the SSD market in 2026.

NicheCommand: A Firehose Becomes a Shortlist

NicheCommand now filters and ranks expired domains, turning a flood into a manageable, prioritized list for buyers and investors.

Announcing Ranveer Singh As Brand Ambassador For Ray-Ban And Ray-Ban Meta In India Along With Exciting New Updates To Our AI Glasses

Meta names Ranveer Singh ambassador for Ray-Ban and Ray-Ban Meta in India as it launches Gen 3 glasses from INR 44,300 and previews new features.