Mixture of experts

An architecture where only a fraction of the network activates for any given token.

Also known as MoE, sparse model

Rather than running every parameter for every token, a router sends each token to a small subset of specialised sub-networks. Among models that publish their configurations, active parameters commonly run at a few per cent of the total.

This is why parameter count has become a poor guide to cost. A model can hold well over a trillion parameters while using only tens of billions per token.