Can Harness Adapters Make Your AI Model More Efficient?
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Can Harness Adapters Make Your AI Model More Efficient? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face says reinforcement-learning LoRA adapters tailored to Pi, OpenCode and Hermes Agent improved Qwen3.8-27B’s results on two benchmarks per harness. The detailed Pi result on Terminal-Bench 2.1 rose from 39% to 55% pass@1 while average token use fell from 2.9K to 1.5K per task; the other comparisons are not quantified in the supplied account. The results are project-team experiments, and independent replication is not reported.

Hugging Face says custom reinforcement-learning LoRA adapters improved the coding results of Qwen3.8-27B when it ran through Pi, OpenCode and Hermes Agent. In the detailed Pi test, the team reports that pass@1 rose from 39% to 55% as average use fell from 2.9K to 1.5K tokens per task, a result that could matter to developers weighing agent performance against token use.

The reported comparison concerns Terminal-Bench 2.1, using Qwen3.8-27B with the Pi agent harness. The project team says the base model scored 39% pass@1 and used an average of 2.9K tokens per task. With a custom reinforcement-learning LoRA adapter, the model scored 55% pass@1 and averaged 1.5K tokens. Pass@1 records whether a task is completed successfully on the first attempt.

The team says it trained separate adapters for Pi, OpenCode and Hermes Agent, and that each outperformed the base model on both of two benchmarks for its matching harness. The supplied account gives detailed numbers only for Pi and does not identify the second benchmark in its summary, so results for the other harnesses cannot be compared numerically here.

According to the project account, the model’s weights stayed frozen while the LoRA adapter was added. The experiments ran on Hugging Face Hub infrastructure using Jobs, TRL’s AsyncGRPOTrainer, OpenEnv and Harbor. The team says it used the harnesses in their shipped form, with a proxy recording tokens, rather than substituting a reimplementation of each agent loop.

At a glance
reportWhen: Reported in the supplied project accoun…
The developmentHugging Face has reported that harness-specific reinforcement-learning LoRA adapters improved Qwen3.8-27B’s coding-agent performance and reduced token use in its evaluations.
AI Agents · Briefing

Harness adapters tune models to the way agents work

Hugging Face reports that harness-specific LoRA adapters improved Qwen3.8-27B coding results while reducing token use. The detailed comparison provided is for Pi.

55%
Pi pass@1, up from 39%
1.5K
average tokens per task, down from 2.9K
3
harnesses: Pi, OpenCode, Hermes Agent

One adapter for each workflow

The team trained adapters inside the harnesses where agents use tools and complete tasks.

Pi

Detailed result reported

On Terminal-Bench 2.1, pass@1 rose from 39% to 55% with the adapter.

OpenCode

Harness-specific tuning

The project says its adapter beat the base model, but supplied figures do not quantify the result.

Hermes Agent

Same reported approach

A separate adapter was trained for this harness; detailed scores were not provided.

What changed in the Pi test

The team attributes the gains to less reasoning, fewer tool calls and more successful first attempts.

Base model
  • 39% pass@1
  • 2.9K average tokens per task
With LoRA adapter
  • 55% pass@1
  • 1.5K average tokens per task

A recipe others can inspect

The project describes training in the shipped agent environments and says artifacts are available on Hugging Face Hub.

1
Prepare coding tasks

Tasks came from existing sources and generated examples, with validation checks described by the team.

2
Train in each harness

The model weights stayed frozen in the reported Pi comparison while a custom LoRA adapter was added.

3
Share artifacts

Adapters, GGUF files, tasks and a SafeAgents plugin are reported as available for inspection.

What the evidence leaves open

The reported gains are tied to the project’s experiments and evaluation setup.

  • Numerical comparisons for OpenCode and Hermes Agent are not included.
  • The supplied account does not provide uncertainty ranges or independent replication.
  • More results are needed to assess the merged “super adapter” and performance on other task sets.
CaveatLower token use may matter for inference costs, but the material does not report costs or show whether the gains generalize.
The takeaway
Harness-specific tuning is promising: the Pi result pairs higher first-attempt success with lower token use, while broader claims need full results and independent replication.

Lower Token Use in Coding Agents

If the reported Pi result holds in other settings, it points to a way to improve an agent’s performance without changing the underlying model weights: train an adapter for the specific harness and workflow in which the model will operate. Since harnesses shape how agents use tools and attempt tasks, results from a general model evaluation may not predict performance inside a particular agent system.

The reported reduction from 2.9K to 1.5K tokens per task could also matter to teams running coding agents repeatedly. Token use is one component of inference cost, and fewer tokens may reflect shorter reasoning or fewer tool interactions. But the project account does not provide cost estimates, and token reductions alone do not establish lower total deployment costs or improved outcomes across all tasks.

The team also says it merged the three adapters into a single “super adapter” intended to run across all three harnesses. If that approach performs reliably, it could simplify distribution. The available summary does not describe the merging method or provide results comparing the merged adapter with the separate versions.

Products Worth Considering

Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation

Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation

Apple MacBook Air 15-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, Midnight

Apple MacBook Air 15-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, Midnight

As an affiliate, we earn on qualifying purchases.

Training Adapters for Each Harness

The work focuses on a practical difference in agent development: a model interacts with tools through a harness, which governs parts of the task workflow. Hugging Face’s project team argues that training inside the target environment can better tune behavior to that workflow than relying only on a model’s general capabilities. The current experiments apply that idea to software-engineering tasks using a 27-billion-parameter model and three agent harnesses.

For training, the team assembled coding tasks in Harbor format from existing datasets and newly generated examples. The account mentions SWE-smith, which introduces bugs into Python and Go repositories, and SWE-Gym tasks packaged in TaskTrove and based on real GitHub issues and merged fixes. It also describes using Tasksmith to create tasks from merged pull requests. The team reports that 24 tasks drawn from 30 pull requests passed checks across 10 Python repositories.

The project says tasks were screened using OpenEnv validation and checked for difficulty against the base model. Its training images included Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6. The account describes this work as an extension of earlier multi-harness reinforcement-learning experiments, but the supplied material does not give a full dataset inventory or complete validation results.

““Most of this is down to the model reasoning less, calling less tools, and performing better first time.””

— Hugging Face project team

How Broad Are the Gains?

The evidence detailed in the supplied account is limited. It reports one numerical comparison, for Pi on Terminal-Bench 2.1, but does not include scores for OpenCode or Hermes Agent or identify the second benchmark in its summary. Sample sizes, uncertainty ranges and task-by-task results are also not provided there.

It is not clear whether the merged adapter matches the performance of each harness-specific adapter, or whether the reported improvements carry over to different models, task sets or deployment conditions. The account does not provide an independent audit of the task verifiers or report independent replication. The figures should therefore be treated as the project team’s experimental findings, not as a guarantee of similar gains for other developers.

Artifacts Available for Testing

Hugging Face says the adapters, GGUF files, reinforcement-learning tasks and SafeAgents plugin are available through a collection on its Hub. Those materials give other researchers and developers a way to inspect the artifacts and attempt the training recipe with their own evaluations.

The next useful evidence would include the full scores for all three harnesses and both benchmarks, details of the adapter-merging process, and independent tests on additional tasks. Until those results are available, the Pi figures are the clearest measure of the reported gains, while the broader performance and efficiency claims remain project-reported.

Key Questions

What did the Pi test report?

Hugging Face reports that Qwen3.8-27B scored 55% pass@1 with a custom LoRA adapter on Terminal-Bench 2.1, compared with 39% for the base model. Average token use was reported as 1.5K per task with the adapter and 2.9K without it.

What does pass@1 mean?

Pass@1 measures whether the model completes a task successfully on its first attempt. It is a task-success measure, not a direct measure of cost or the quality of every part of a solution.

Did the adapters improve all three agent harnesses?

The project team says adapters for Pi, OpenCode and Hermes Agent each beat the base model on two benchmarks under their matching harness. The supplied account gives numerical details only for Pi, so the size of the other reported gains cannot be assessed from it.

Have the results been independently verified?

The supplied material does not report independent replication. Hugging Face says relevant adapters, tasks and other artifacts are available on its Hub, which may let others test the findings.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Can We Trust AI? Anthropic’s Invisible Watermark Identifies Chatbot Creations

Anthropic has added an invisible watermark to Claude-generated content, aiming to verify AI origin, but technical details and detection methods remain unclear.

Metal Gear Solid 4 (2008) still holds the record for the longest cutscene in video game history, with one continuous cutscene lasting for 71 minutes

Metal Gear Solid 4 (2008) still holds the record for the longest uninterrupted cutscene in video game history at 71 minutes.

Ploopy Bean: a trackpoint for every computer

Ploopy launches the Bean Pointing Stick, a fully assembled, open-source trackpoint compatible with any setup, shipping from May 2026.

The United Kingdom: The Pragmatist’s Hedge

Post-Brexit UK adopts a cautious, flexible model emphasizing work incentives, light AI regulation, and moderate welfare support amid evolving economic challenges.