Robot failure monitors are usually judged on how cleanly they separate failed runs from successful ones. A paper posted to arXiv argues that this misses the two costs that matter once a robot is deployed: an alarm too late for anything to be recovered, and an intervention that disrupts a run which would have succeeded anyway.
The framework, RESCUE, restores simulator and policy state at checkpoints during a run, tries alternative actions from that point, and asks whether acting on an alarm lowers the expected cost of finishing the task. The authors applied it to two vision-language-action policies, OpenVLA-OFT and pi-0.5, on the LIBERO-Spatial and LIBERO-Plus benchmarks, running 128,340 counterfactual branches.
The headline result is a tolerance figure: under a stop-and-reset response, even an idealised detector can afford a false-alarm rate of only about 2.0 to 3.3% before unnecessary resets cancel out the benefit of catching failures early. That is a tight budget for systems whose published scores often look comfortable.
They also report that recoverability depends on the pairing of robot state and available intervention, not the state alone. In one condition, 14 of 66 failures had a recoverable checkpoint if the robot carried on, rising to 34 of 66 when a scripted recovery was available. This is a preprint, and the numbers are the authors’ own.