A new benchmark tests whether agents can turn an expert’s plan into a fast GPU kernel

D2K-Bench, published through Alibaba’s Qwen GitHub organisation, runs 26 kernel-writing tasks twice, with and without expert design guidance, and finds the guidance roughly doubles the speedup the best models achieve.

6 Oct 20261 min readHardwareResearchChina

A benchmark released through Alibaba’s Qwen GitHub organisation tests a narrower question than whether a model can write a GPU kernel. D2K-Bench, described in a paper posted to arXiv on 2 October, gives agents 26 tasks and 85 workloads and measures how well they turn expert design guidance into fast, correct kernels.

The guidance comes in three layers: high-level algorithmic insight, dataflow design, and low-level optimisation tricks. Each task runs twice, with and without it, holding the task description, workloads, tools, hardware and a 350-turn budget constant, so the comparison is paired rather than inferred.

On Nvidia B200 GPUs across five models, the guidance lifted correctness over 130 model-task pairs from 93.1% to 98.5%, and the paper’s performance score across all 26 tasks from 1.46 to 1.95. For the three models that submitted correct code on every task in both runs, which the paper names as GPT-6-Astra, Claude Opus 4.8 and GPT-5.6-Sol, geometric mean speedup rose from 1.69 times to 2.49 times.

Read that as a result about where the gap sits. Handed an expert’s plan, agents produce substantially faster code, which suggests the weaker link is working out what to do rather than writing it. The paper also flags design properties that go unimplemented even when spelled out. All figures are the authors’ own.

Primary source

the D2K-Bench paper.


Related