🔍 Read the full analysis: FrontierCode: Would A Maintainer Merge Your AI’s Pull Request? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, and tested 23 models on repository tasks judged against maintainer-oriented criteria. Claude Opus 5.5 led the results, but cleared all blocking checks in 57.7% of attempts; cost and performance varied widely, and several models had no result.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark designed to assess whether code changes would satisfy repository checks and maintainer-style review criteria. Claude Opus 5.5 led with a 52.2% score, but it passed every blocking check in 57.7% of attempts, leaving substantial room for human review before AI-generated pull requests can be treated as reliably mergeable.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026. Its test used the benchmark’s 100-task Main set and a common open-source agent harness, Stirrup. Each model had up to 250 turns per attempt and three attempts per task. The reported leaderboard score is distinct from the pass rate: the score reflects the benchmark’s graded results, while the pass rate records attempts that cleared every blocking check.
GPT-6.1 Sol scored 49.1%, just ahead of Claude Fable 5.1 at 49.0%. Artificial Analysis reported that Sol cost $0.81 per task, compared with $10.15 for Opus. Xiaomi’s MiMo-V2.6-Pro scored 45.4%, making it the highest-scoring open-weights model in the reported results. The source says nine of the benchmark’s 100 tasks were not solved by any model on any attempt.
FrontierCode’s checks cover hidden tests, command checks for regressions and cleanliness, tests that must fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM code review against maintainer criteria. Failing any blocking check earns zero; passing them all produces a score from 70% to 100%, depending on quality checks. The benchmark also assigns zero to attempts flagged by Cognition’s trajectory scanner for looking up solutions, according to the source account.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
The results distinguish functional correctness from a patch a maintainer would accept. A change can pass ordinary tests yet still introduce regressions, touch too many files, add weak tests, or fail other repository-specific standards. FrontierCode attempts to capture those concerns, making its results relevant to teams deciding how much review AI-written changes need.
The scores do not establish that a model will fail at the same rate in every workplace or repository. They do show that, on this benchmark, even the top model did not clear all blocking checks in more than 57.7% of attempts. That limits what can be inferred from claims that coding agents can independently produce ready-to-merge changes. For maintainers, the practical implication is that human review remains part of the workflow, while model choice may depend on the trade-off between score and cost.
That trade-off is pronounced in the reported figures. Artificial Analysis listed GPT-6 Luna at 41.9% for $0.09 per task, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus at 52.2% for $10.15. Moving from Sol to Opus adds 3.1 percentage points in score at about 12.5 times the reported task cost. These are benchmark-specific figures, not a guarantee of equivalent savings or outcomes in production.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Judges Patches
FrontierCode was developed by Cognition with open-source maintainers, according to the source material. It was announced in June 2026, followed by version 1.1 in July, which introduced rules against looking up solutions online. More than 20 maintainers created tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The full benchmark contains 150 tasks, while Artificial Analysis ran its 100-task Main set; the tasks are private to reduce the chance models encountered them during training.
Artificial Analysis used GPT-6 Luna (medium) as the judge for the LLM review portion. Its results show a tight top group: nine models from six labs scored between 43% and 52%. Below that group, the reported scores fell to 33.8% for Grok 4.7, 23.0% for MiniMax-M3, 15.8% for Inkling, 13.9% for Muse Glimmer and 10.5% for Nemotron 3 Ultra. The benchmark’s combination of blocking checks and quality scoring is intended to penalize patterns that can make a patch harder to review, not just code that fails tests.
Artificial Analysis also compared the results with its Intelligence Index. In that comparison, Claude Sonnet 5.5 ranked second among the listed models on the general index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. DeepSeek V4.1 Flash and GPT-6 Luna performed better on FrontierCode relative to their general-index positions. General benchmark rankings did not map directly to patch-review performance in the comparison provided.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
Models and Real-World Fit
The results cover a defined benchmark setup, not all programming languages, repositories, teams or production workflows. The source does not establish how closely the private task set represents the pull requests maintainers receive day to day, or how scores would change under different agent tools, review systems or task budgets. The leaderboard should be read as a comparison under Artificial Analysis’s stated setup, rather than a direct forecast of workplace acceptance rates.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published results do not provide a complete comparison across all models mentioned in the source. The material also does not give confidence intervals or other uncertainty estimates for the scores, so the small gaps near the top—such as 49.1% versus 49.0%—should not be treated as proof of a meaningful performance difference.
More Results Could Shift Rankings
The immediate next point of interest is whether models missing from the October 13 results receive FrontierCode evaluations. In particular, the source identifies Muse Wonder Full as a model whose result readers may watch, based on its performance on other coding evaluations; no FrontierCode score for it was available at that time. Any comparison will depend on results being produced under comparable benchmark and harness conditions.
For developers and maintainers, the next practical step is to test candidate models against their own repositories and review standards rather than treating a leaderboard score as a merge decision. Further benchmark updates could clarify whether rankings hold across additional tasks or model versions. Until then, the published evidence supports a limited conclusion: FrontierCode rewards more than passing tests, and its leading model still failed to clear a blocking check in a substantial share of attempts.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated changes to real repository tasks using checks for correctness, regressions, test quality, scope and an LLM review against maintainer criteria. Its goal is to assess whether a patch meets more than basic test-passing requirements.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s reported results with a 52.2% FrontierCode score and a 57.7% pass rate for attempts clearing every blocking check.
Does the top score mean AI pull requests are ready to merge?
No. The reported result is a benchmark score, not a guarantee that a patch should be merged. The top model cleared all blocking checks in 57.7% of attempts, and repository teams still need to review changes in their own context.
Which open-weights model scored highest?
MiMo-V2.6-Pro was the highest-scoring open-weights model listed, at 45.4%. DeepSeek V4.1 Flash scored 44.0%, and Kimi K3 scored 43.6%.
Were all major models included?
No. As of October 13, 2026, the source reported no FrontierCode result for Gemini 4 Argon, Muse Wonder, Muse Wonder Full or Mistral Large 4 Preview. Their absence leaves the comparison incomplete.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
![Rails World 2026 Opening Keynote [Video]](https://1023jack.com/wp-content/uploads/2026/09/rails-world-2026-opening-keynote-video-featured-560x300.jpg)
