What it is. Total parameters is the size of the whole model; active parameters is how much of it actually runs for each piece of text it processes.
An analogy. A large hospital might employ hundreds of specialists, but you see only a couple of them for any one visit. The hospital’s size (total staff) and the cost of your visit (the doctors you actually see) are two different numbers.
How it works. Models built as a mixture of experts contain many smaller sub-networks called experts. For every token, a small “router” picks a few experts to handle it and the rest sit idle. Memory has to hold every expert, so total parameters decide how much hardware you need to load the model. Active parameters decide how much computing each token costs, which drives speed and price.
Why it was in the news this week. Meta’s Muse Spark Open has roughly 816bn total parameters but about 42bn active per token: each of its 62 layers holds 256 experts and selects eight. Nvidia’s Bolt 74B-A9B puts both numbers in its name: 74bn total, 9bn active. The pattern has been common for a while; Mistral’s Mixtral 8x7B, released in December 2023, had 46.7bn total parameters and used 12.9bn per token.
One thing to remember. When comparing models, ask which number is being quoted. A big total can still be fast to run, but it still needs enough memory to hold all of it. That is why Muse Spark, despite using only about 42bn parameters per token, ships as a checkpoint of roughly 1.5TB that most single machines cannot load. A dense 42bn-parameter model would be far easier to host; the mixture-of-experts design trades extra memory for extra capacity.
Sources: AIooze brief on Muse Spark Open; Mistral AI, “Mixtral of experts”, 11 December 2023