🔍 Read the full analysis: A Closer Look At CUGA, Hermes, And Deep Agents For AI Work on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The test suggests harness configuration can affect performance on this benchmark, but does not establish a general product ranking or test policy enforcement.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks, ahead of Hermes at 42 and Deep Agents at 27, with all three systems using the same open-weight model, GLM-5.3-flash. The result matters because it suggests that an agent’s software setup—not only its underlying model—can shape performance on a particular multi-step workload; it does not show that CUGA is generally superior. For context, see the original analysis comparing these harnesses.
The comparison used 90 tasks across 30 scenarios, with one attempt per task, and ran through the Harbor evaluation framework. IBM reports task pass rates of 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. On a separate scenario measure—counting a scenario only when all three of its variants passed—CUGA passed 20 scenarios, Hermes eight and Deep Agents two.
The three systems did not use identical configurations. CUGA used task decomposition, app-focused tools, pre-execution verification and automatic reflection. Hermes ran with Harbor’s default setup and AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP with LangChain middleware for tool selection, context summarization, a to-do list, retries and call limits. IBM also says it made two fixes to Deep Agents so malformed model replies would not end runs.
IBM says the experiment setup is publicly available through harbor-eval, its fork of Harbor. The supplied source describes this as an IBM-reported experiment; it does not provide an independent replication. The shared model controls one important variable, but the different harness designs mean the results do not isolate every factor behind the score gap.
Why Agent Harnesses Affect Results
The reported spread—from 30% to 81.1% task completion with the model held constant—shows why organizations may need to evaluate the full agent configuration, rather than relying on a model score alone. A harness coordinates how a task is broken into steps, which tools are available and whether actions or results are checked. Those design choices can affect whether a multi-step job succeeds.
For teams considering agents, the practical takeaway is to test the model, harness and workflow together on representative tasks. The AppWorld result may be relevant to work involving sequences of API calls across applications, but it is not evidence that CUGA will lead on personal-assistant tasks, research or other workloads for which Hermes and Deep Agents may be used. The experiment also did not test policy enforcement, so the scores do not establish that any system reliably follows enterprise rules.
Products Worth Considering
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Waterproof, Dustproof, USB‑C, Up to 12 hr Battery, Black
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
As an affiliate, we earn on qualifying purchases.
What AppWorld Measures
AppWorld tests multi-step tasks across nine simulated apps and 457 APIs, including email, payments and file storage. IBM characterizes it as a proxy for some abilities relevant to enterprise automation, such as reading API documentation, paging through results and changing state across applications. It uses consumer apps, however, and the supplied material does not establish how results translate to real enterprise systems.
The source describes Hermes, from Nous Research, as designed for always-on personal assistants, and Deep Agents, from LangChain, as suited to longer multi-step work such as research. CUGA—short for ConfigUrable Generalist Agent—is described as IBM’s open-source harness for work across systems such as customer relationship management and payments. IBM characterized its results as “not a verdict on Hermes or Deep Agents.” The comparison is best read as a result on one benchmark, not a broad contest among products.
The source also points to community directories that track nearly 200 agent harnesses and cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro result rose from 23% to 52% when the harness changed. That is a distinct comparison, not part of the AppWorld experiment, and should not be treated as independent confirmation of IBM’s results.
““This is not a verdict on Hermes or Deep Agents.””
— IBM
Limits of the AppWorld Comparison
The reported result covers one model, 90 tasks and one benchmark, with a single attempt per task. It remains unclear whether the relative scores would hold across other models, task sets, repeated runs or production workloads. The harnesses were configured differently, and the supplied material does not show a test that separates the impact of each individual design choice.
The experiment did not use policies and therefore provides no evidence about policy enforcement. IBM describes CUGA features for applying rules through the harness, but their effectiveness is not established by these AppWorld scores. The source also gives no comparative data on cost, latency, large-scale reliability or deployment outcomes, and supplies no timeline for an independent replication or a broader study.
How Teams Can Test Their Workflows
The experiment’s public harbor-eval setup gives readers a way to inspect the reported evaluation. Teams considering these systems can also compare them on workflows that resemble their own, using the models, tools and configurations they expect to deploy. Repeated runs and varied task sets would help show whether a result is consistent rather than tied to a single attempt.
For enterprise use, evaluations would need to cover requirements absent from this benchmark, including access controls, approval steps, data boundaries and performance in the organization’s deployment environment. The supplied material identifies no scheduled follow-up study, so whether IBM or another group will publish broader comparisons remains unknown.
Key Questions
What did IBM report in the AppWorld test?
IBM reported that CUGA completed 73 of 90 tasks with GLM-5.3-flash. Hermes completed 42 and Deep Agents 27 in the same benchmark comparison.
Does the result prove CUGA is the best agent?
No. It is a result from one model and one benchmark, with differently configured harnesses. It does not establish a general ranking across tasks or deployments.
What is an agent harness?
A harness is software that coordinates an AI agent’s work, including how it divides tasks, uses tools and checks actions or results. The experiment suggests these choices can affect task completion on a specific workload.
Did the experiment test enterprise policy enforcement?
No. The supplied material says the experiment did not use policies. The scores do not show whether any system reliably enforces access rules or other enterprise requirements.
Can teams inspect or repeat the setup?
IBM says the setup is publicly available through harbor-eval, its Harbor fork. The source does not report an independent replication or give a timeline for one.
Primary source: Hugging Face · via ThorstenMeyerAI.com
