IBM Research published work on 15 September putting a number on something anyone who has deployed an AI agent will recognise: the same agent, given the same task five times, does not necessarily do it five times. A ReAct agent built on GPT-4.1 succeeded on 77.4% of attempts when scored as an average across five runs. Scored on whether it succeeded on all five runs of a task, it managed 53.0%. The 24.4-point gap is made up of tasks the agent can sometimes do and sometimes cannot, with nothing changed between attempts.
The distinction is between mean@5, which averages over runs, and pass^5, which requires every run to succeed. Published agent benchmark scores are usually closer to the first than the second. That is much of why an agent that looks dependable in evaluation feels erratic in real work.
IBM’s response is a diagnostic tool it calls the Consistency Analyzer. Rather than replaying whole tasks, it resamples each decision point in a recorded run — one extra model call per decision — to find the steps where the agent flips between choices, then writes targeted guidelines for those moments. Applying them raised pass^5 from 53.0% to 69.0% without costing accuracy, with mean@5 moving from 77.4% to 81.0%. The code is published as ALTK-Evolve.
Source: IBM Research.