🔍 Read the full analysis: Can Harness Adapters Make Your AI Model More Efficient? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face says reinforcement-learning LoRA adapters tailored to Pi, OpenCode and Hermes Agent improved Qwen3.8-27B’s results on two benchmarks per harness. The detailed Pi result on Terminal-Bench 2.1 rose from 39% to 55% pass@1 while average token use fell from 2.9K to 1.5K per task; the other comparisons are not quantified in the supplied account. The results are project-team experiments, and independent replication is not reported.
Hugging Face says custom reinforcement-learning LoRA adapters improved the coding results of Qwen3.8-27B when it ran through Pi, OpenCode and Hermes Agent. In the detailed Pi test, the team reports that pass@1 rose from 39% to 55% as average use fell from 2.9K to 1.5K tokens per task, a result that could matter to developers weighing agent performance against token use.
The reported comparison concerns Terminal-Bench 2.1, using Qwen3.8-27B with the Pi agent harness. The project team says the base model scored 39% pass@1 and used an average of 2.9K tokens per task. With a custom reinforcement-learning LoRA adapter, the model scored 55% pass@1 and averaged 1.5K tokens. Pass@1 records whether a task is completed successfully on the first attempt.
The team says it trained separate adapters for Pi, OpenCode and Hermes Agent, and that each outperformed the base model on both of two benchmarks for its matching harness. The supplied account gives detailed numbers only for Pi and does not identify the second benchmark in its summary, so results for the other harnesses cannot be compared numerically here.
According to the project account, the model’s weights stayed frozen while the LoRA adapter was added. The experiments ran on Hugging Face Hub infrastructure using Jobs, TRL’s AsyncGRPOTrainer, OpenEnv and Harbor. The team says it used the harnesses in their shipped form, with a proxy recording tokens, rather than substituting a reimplementation of each agent loop.
Harness adapters tune models to the way agents work
Hugging Face reports that harness-specific LoRA adapters improved Qwen3.8-27B coding results while reducing token use. The detailed comparison provided is for Pi.
One adapter for each workflow
The team trained adapters inside the harnesses where agents use tools and complete tasks.
Detailed result reported
On Terminal-Bench 2.1, pass@1 rose from 39% to 55% with the adapter.
Harness-specific tuning
The project says its adapter beat the base model, but supplied figures do not quantify the result.
Same reported approach
A separate adapter was trained for this harness; detailed scores were not provided.
What changed in the Pi test
The team attributes the gains to less reasoning, fewer tool calls and more successful first attempts.
- 39% pass@1
- 2.9K average tokens per task
- 55% pass@1
- 1.5K average tokens per task
A recipe others can inspect
The project describes training in the shipped agent environments and says artifacts are available on Hugging Face Hub.
Tasks came from existing sources and generated examples, with validation checks described by the team.
The model weights stayed frozen in the reported Pi comparison while a custom LoRA adapter was added.
Adapters, GGUF files, tasks and a SafeAgents plugin are reported as available for inspection.
What the evidence leaves open
The reported gains are tied to the project’s experiments and evaluation setup.
- Numerical comparisons for OpenCode and Hermes Agent are not included.
- The supplied account does not provide uncertainty ranges or independent replication.
- More results are needed to assess the merged “super adapter” and performance on other task sets.
Lower Token Use in Coding Agents
If the reported Pi result holds in other settings, it points to a way to improve an agent’s performance without changing the underlying model weights: train an adapter for the specific harness and workflow in which the model will operate. Since harnesses shape how agents use tools and attempt tasks, results from a general model evaluation may not predict performance inside a particular agent system.
The reported reduction from 2.9K to 1.5K tokens per task could also matter to teams running coding agents repeatedly. Token use is one component of inference cost, and fewer tokens may reflect shorter reasoning or fewer tool interactions. But the project account does not provide cost estimates, and token reductions alone do not establish lower total deployment costs or improved outcomes across all tasks.
The team also says it merged the three adapters into a single “super adapter” intended to run across all three harnesses. If that approach performs reliably, it could simplify distribution. The available summary does not describe the merging method or provide results comparing the merged adapter with the separate versions.
Products Worth Considering

Samsung Galaxy Tab A11+ Tablet (6GB RAM, 128GB Storage, Gray)

Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation

Apple MacBook Air 15-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, Midnight
As an affiliate, we earn on qualifying purchases.
Training Adapters for Each Harness
The work focuses on a practical difference in agent development: a model interacts with tools through a harness, which governs parts of the task workflow. Hugging Face’s project team argues that training inside the target environment can better tune behavior to that workflow than relying only on a model’s general capabilities. The current experiments apply that idea to software-engineering tasks using a 27-billion-parameter model and three agent harnesses.
For training, the team assembled coding tasks in Harbor format from existing datasets and newly generated examples. The account mentions SWE-smith, which introduces bugs into Python and Go repositories, and SWE-Gym tasks packaged in TaskTrove and based on real GitHub issues and merged fixes. It also describes using Tasksmith to create tasks from merged pull requests. The team reports that 24 tasks drawn from 30 pull requests passed checks across 10 Python repositories.
The project says tasks were screened using OpenEnv validation and checked for difficulty against the base model. Its training images included Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6. The account describes this work as an extension of earlier multi-harness reinforcement-learning experiments, but the supplied material does not give a full dataset inventory or complete validation results.
““Most of this is down to the model reasoning less, calling less tools, and performing better first time.””
— Hugging Face project team
How Broad Are the Gains?
The evidence detailed in the supplied account is limited. It reports one numerical comparison, for Pi on Terminal-Bench 2.1, but does not include scores for OpenCode or Hermes Agent or identify the second benchmark in its summary. Sample sizes, uncertainty ranges and task-by-task results are also not provided there.
It is not clear whether the merged adapter matches the performance of each harness-specific adapter, or whether the reported improvements carry over to different models, task sets or deployment conditions. The account does not provide an independent audit of the task verifiers or report independent replication. The figures should therefore be treated as the project team’s experimental findings, not as a guarantee of similar gains for other developers.
Artifacts Available for Testing
Hugging Face says the adapters, GGUF files, reinforcement-learning tasks and SafeAgents plugin are available through a collection on its Hub. Those materials give other researchers and developers a way to inspect the artifacts and attempt the training recipe with their own evaluations.
The next useful evidence would include the full scores for all three harnesses and both benchmarks, details of the adapter-merging process, and independent tests on additional tasks. Until those results are available, the Pi figures are the clearest measure of the reported gains, while the broader performance and efficiency claims remain project-reported.
Key Questions
What did the Pi test report?
Hugging Face reports that Qwen3.8-27B scored 55% pass@1 with a custom LoRA adapter on Terminal-Bench 2.1, compared with 39% for the base model. Average token use was reported as 1.5K per task with the adapter and 2.9K without it.
What does pass@1 mean?
Pass@1 measures whether the model completes a task successfully on its first attempt. It is a task-success measure, not a direct measure of cost or the quality of every part of a solution.
Did the adapters improve all three agent harnesses?
The project team says adapters for Pi, OpenCode and Hermes Agent each beat the base model on two benchmarks under their matching harness. The supplied account gives numerical details only for Pi, so the size of the other reported gains cannot be assessed from it.
Have the results been independently verified?
The supplied material does not report independent replication. Hugging Face says relevant adapters, tasks and other artifacts are available on its Hub, which may let others test the findings.
Primary source: Hugging Face · via ThorstenMeyerAI.com
