TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A live experiment by Firmulate tested AI models in managing a small business during its worst week. Results show models excel at diagnosis but struggle with execution and trust, shifting focus from chat quality to management skills. This new benchmark could reshape AI evaluation for enterprise use.
Firmulate has conducted a groundbreaking live experiment where AI models manage a simulated small business during its most challenging week. The test evaluates not just chat responses but the models’ ability to diagnose, decide, communicate, and complete tasks under real operational pressures. The results highlight a significant gap in current AI benchmarks, emphasizing management quality over conversational prowess, and suggest a new direction for enterprise AI evaluation.
The experiment involved five AI managers competing in the July 2026 Crucible League. The models were scored on their ability to handle crises, negotiate deals, and maintain trust within a simulated company burning €105,000 monthly against €2,300 MRR. For more on this innovative testing approach, see the original analysis. The top performer, gpt-5.6-sol, achieved a score of 95, while others lagged behind, with Opus 4.8 at 73. Despite all models identifying crises and resisting manipulation, only two successfully signed a €55,000 deal, demonstrating that diagnosis alone does not guarantee execution. The experiment also revealed that models could sound informed but still omit critical facts necessary for decision-making, impacting business outcomes.
Safety and trust were prioritized, with a strict cap on breaches—any breach ended the evaluation. All models refused social engineering attempts, such as fake CEO messages or background approvals, showing strong resistance to manipulation. However, even the most thorough model, Opus 4.8, failed to escalate issues into the proper channels, illustrating that effort and complexity do not necessarily translate into effective management. The competition’s design included varied operational parameters; for example, Kimi K3 used default API settings, which contextualizes its performance relative to others.
The live company simulated 13 synthetic employees and real money mechanics, with a monthly burn rate of €105,000. It employed 680+ self-learned rules and versioned workdays, transforming management into an observable process rather than a static response. This setup allowed testing whether AI agents can prioritize, read organizational context, resist shortcuts, and preserve trust over multiple days and decisions. The real-world relevance is underscored by the 242 unedited management decisions behind the experiment, illustrating the difficulty of attributing success solely to language quality. Insights from this study are discussed in the original analysis.
Why Management Performance Matters in AI Benchmarks
This experiment shifts the focus of AI evaluation from chat or coding benchmarks to real-world management capabilities, emphasizing trust, execution, and decision-making under pressure. For enterprises, this means assessing whether AI can genuinely handle operational responsibilities, not just produce convincing responses. The findings suggest that current benchmarks may overvalue superficial performance, while neglecting critical management skills like escalation, honesty, and strategic execution. As AI models move into roles requiring accountability, these management-focused metrics could become essential, influencing how organizations select and trust AI assistants for complex, consequential tasks.
As an affiliate, we earn on qualifying purchases.
The Evolution of AI Evaluation Toward Management Skills
Traditional AI benchmarks have focused on language proficiency, coding accuracy, or chatbot engagement, often measured in isolated tests or competitions. However, as AI begins to take on more operational roles, the limitations of these benchmarks become apparent. The Firmulate experiment builds on prior discussions about AI’s capabilities in real-world settings, highlighting that diagnosis alone is insufficient. Managing crises, making decisions under constraints, and maintaining trust are complex skills that current benchmarks do not adequately capture. The July 2026 Crucible League represents a step toward evaluating AI in scenarios that mirror actual business challenges, emphasizing the importance of holistic management performance.
Previous efforts, such as coding leaderboards or chat arenas, have shown progress but remain narrow in scope. This new approach, testing models in a live company environment with real consequences, aims to close the gap between AI’s technical proficiency and its practical utility in enterprise settings. The experiment’s design reflects growing recognition that trustworthiness, execution, and organizational awareness are critical for deploying AI in high-stakes roles.
“The real test of AI management isn’t just whether it can diagnose a problem, but whether it can follow through, escalate appropriately, and preserve trust under pressure.”
— Thorsten Meyer, Lead Researcher at Firmulate
Unanswered Questions About Long-Term AI Management Capabilities
It remains unclear how well these models would perform in longer-term, real-world business operations beyond the controlled experimental setting. The experiment focused on a single week of crises, and the models’ ability to sustain trust, adapt to evolving scenarios, or handle unforeseen complications over months is still untested. Additionally, the impact of different organizational structures, industries, or company sizes on AI management performance has not been explored. The extent to which these benchmarks predict actual enterprise success remains an open question, as does how future models will evolve to meet these management demands.
Next Steps for AI Management Benchmarking and Adoption
Following the July 2026 results, firms and AI developers are expected to refine evaluation methods, emphasizing management and trust metrics. Firms considering AI for operational roles will likely run their own simulations and wargames, similar to Firmulate’s approach, to assess real-world readiness. Researchers will explore extending these benchmarks to longer periods and more complex scenarios, aiming to better predict AI’s capacity to manage organizational consequences over time. Meanwhile, industry standards may evolve to incorporate management-focused assessments as a core component of AI deployment decisions, shaping the future landscape of enterprise AI adoption.
Key Questions
How does this new benchmark differ from traditional AI tests?
This benchmark evaluates AI models in managing a simulated company through crises, focusing on decision-making, trust, escalation, and execution, rather than just language or coding accuracy.
Why is management performance more important than chat quality?
Management performance directly impacts organizational trust, operational success, and risk mitigation, making it crucial for AI to handle real-world responsibilities effectively.
Can current AI models handle real business operations?
They show promise in diagnosis and resisting manipulation but still struggle with execution, escalation, and maintaining trust over complex, multi-day scenarios.
What are the limitations of this experiment?
It covers a single week of simulated crises and does not address long-term management, evolving scenarios, or industry-specific challenges.
What will happen next in AI management evaluation?
Expect more comprehensive benchmarks, real-world simulations, and industry adoption of management-focused metrics to better assess AI’s operational readiness.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
