The vocabulary is not designed by hand. Algorithms such as byte-pair encoding repeatedly merge the most frequent adjacent pair of symbols until a target vocabulary size is reached, so frequency in the training data determines what gets a clean single token.
The consequence is that tokenisation encodes the biases of its corpus. Languages under-represented in training fragment more aggressively, costing their speakers more tokens for the same meaning.
