The Allen Institute for AI published a detailed account on 9 October of replacing the priority-based scheduler on its research GPU clusters with one built around allocated time budgets. Managers hand each research programme a share of GPU time, every request for protected capacity draws down that budget, and a hierarchical fair-share queue favours whichever allocation is currently under-served across a rolling seven-day window. Jobs declare a minimum runtime, capped at eight hours, during which they cannot be pre-empted. Anything declaring zero runs on unallocated, always-pre-emptible capacity.
The hardware involved is thousands of NVIDIA H100, B200 and B300 GPUs, split into pools of 88 to 1,024 and shared by roughly 150 researchers, with outstanding requests running at two to three times the GPUs available.
The numbers are worth reading carefully. Occupancy sat at 98% before the change and 98% after, so this is not a utilisation story. What moved was waiting. The 90th-percentile wait for small debug jobs fell from two hours to 30 seconds, and the median queue wait on the largest H100 pool from five minutes to 24 seconds. Teams received 98% of the GPU hours they were owed over a 30-day test period, with 13 of 15 allocations getting at least 95%. The post also quotes a researcher saying it feels like an extra 30% of compute, which is an impression rather than a measurement.
Ai2 has written the change up in full on its Hugging Face blog.