AI can draft a pitch in seconds. But would it recognize the deal its own analysis supports, hold its nerve under pressure, and follow company rules when a crisis hits? A live Firmulate experiment puts AI models in charge of a small software company to find out. The test is watchable at Firmulate.
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A tougher test than a chat demo
For its final Crucible League, in July 2026, Firmulate gave frontier AI models the same small software company and its worst week: the same customers, crises and temptations. Every decision was versioned and auditable. The point was to see how models handled management decisions, not just how convincing they sounded in a conversation.
The results put gpt-5.6-sol first with 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. Under the league’s trust rule, a single breach caps the total: “no amount of good work outweighs a breach of trust.”
Seeing the problem was not enough
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal that their own analysis had earned. The experiment’s phrase for that divide was “Same diagnosis, same pitch — no signature.” A polished answer, in other words, did not guarantee the model would carry its recommendation through to the decision.
The deal hinged on a detail that was easy to miss: the decisive competitor weakness appeared two document references deep in the company’s own files, not in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. The difference came down to whether the model followed evidence far enough to make the case—and then closed.
Pressure tests and uneven performance
Firmulate also tested social engineering with fake CEO messages that escalated over three stages, followed by a reporter asking for “just one yes/no, on background.” All five models refused. Kimi K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Opus 4.8 offers a more complicated picture than its fifth-place finish suggests. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses. But it left the close on the table, and discipline slipped when it tried to write into a locked department instead of escalating. A weaker version of the same issue appeared in all four models.
There is a fairness detail in the comparison: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That context matters when reading the rankings. The league is one reported test, not a promise about how a model will behave in every company or deployment.
A company you can watch
The live Firmulate company has 13 synthetic employees and real money mechanics: it burns €105k per month against €2.3k MRR, with a public cash countdown. Its workers learn from experience; the site reports 680+ self-learned playbook rules, and every workday is versioned. Firmulate also offers a quiz built from 242 real, unedited management decisions: readers can guess which model made each call.
Those details make the experiment more than a leaderboard. Visitors can follow an operating company with mounting financial pressure and see decisions accumulate over time. For technology readers accustomed to AI demos built around a single prompt, the live company presents a different question: how does a model behave when decisions have consequences across a difficult stretch of work?
From watching to a company-specific pilot
For businesses, the next step is to try the wargame on their own operating context. Firmulate says enterprises can provide a read-only export of their business and run crisis scenarios against it, producing a board report with model rankings and weak points in their playbooks. Nothing writes back to real systems. The aim is to learn how models handle a company’s customers, rules and pressure before giving them a role in live operations.
The Crucible results suggest that spotting a crisis and refusing a scam are only part of the job. A model also has to find the relevant evidence, act on its analysis and respect operating boundaries. Firmulate’s live experiment lets readers watch those choices unfold; an enterprise pilot applies the exercise to a company’s own data and scenarios.
To discuss a pilot using a read-only export of your business, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
