A new benchmark checks whether the database changed, not whether the agent said it did

Microsoft and Hugging Face have released ThinkingBox, which finds that most agent runs failing an executable database check still finish cleanly and report no error.

6 Oct 20261 min readApplicationsResearchUnited States

Microsoft and Hugging Face have published ThinkingBox, a benchmark that scores AI agents on whether a database actually changed rather than on whether the agent reported finishing. The headline finding is the gap between the two.

Across 507 stateful business workflows, each run 20 times against a range of language models, the authors report that of the runs that failed the executable state check, “67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error”. The agent looked successful by every available signal short of inspecting the data.

On pass@1, Claude Opus 5.5 leads overall at 67.16%. The consistency numbers are worth more than the leaderboard position: the authors note that Kimi-K3 solves 75 more tasks at least once than Opus 5, while Opus 5 solves 173 more tasks consistently than Kimi-K3. Breadth and reliability are not the same property, and a buyer piloting an agent on a handful of runs will mostly measure the first.

The tasks cover retail (98), auto insurance (100), travel (104), a neobank (104) and consulting (101). The harness is MIT-licensed, the benchmark data is released under CDLA-Permissive-2.0 and the OpenEnv environment under BSD-3-Clause, so the figures can be argued with rather than taken on trust.

Primary source

Microsoft and Hugging Face, The Agent Said It Was Done. The Database Disagreed.


Related