Rather than running every parameter for every token, a router sends each token to a small subset of specialised sub-networks. Among models that publish their configurations, active parameters commonly run at a few per cent of the total.
This is why parameter count has become a poor guide to cost. A model can hold well over a trillion parameters while using only tens of billions per token.
