HBM is memory built as a vertical stack of chips, mounted next to the processor and connected by very many wires, so data moves far faster than with ordinary memory. It is a standard set by JEDEC, the electronics standards body, now in its fourth generation, HBM4.
Because running AI models is often limited by how fast weights can be fetched, HBM capacity and bandwidth are headline specs for AI accelerators. Microsoft’s Maia 300, for example, carries eight HBM4 stacks with 18.0 TB/s of peak bandwidth.