A language model never sees a word. It sees a number. The translation between the two happens in two steps — tokenisation and embedding — and a surprising share of AI’s stranger failures become obvious once you understand them.
Tokens: chopping text into pieces
Before a model can process text, the text is split into tokens — the atomic units of its vocabulary. Tokens are usually not words. They sit between characters and words, typically averaging around three to four characters of English.
Why this awkward middle ground? Character-level vocabularies are tiny but make sequences very long, which is expensive given that attention cost grows quadratically. Word-level vocabularies produce short sequences but cannot handle anything unseen — every typo, name and neologism becomes an unknown. Subword tokenisation is the compromise: common words get a single token, rarer words break into reusable fragments, and nothing is ever truly out of vocabulary.
The vocabulary is not designed by hand. It is learned from a corpus by algorithms such as byte-pair encoding, which repeatedly merges the most frequent adjacent pair until a target vocabulary size is reached. Frequency in the training corpus therefore determines what gets a clean single token.
What this causes
- Letter-level tasks are genuinely hard. Asking a model to count the letters in a word, or reverse it, asks it to inspect the interior of units it does not natively decompose. This is the real source of a well-known class of embarrassing failures — not stupidity, but a representational blind spot.
- Arithmetic is structurally awkward. Numbers tokenise inconsistently, so digits that should align positionally may not be separate units at all.
- Languages are not priced equally. Tokenisers trained on predominantly English corpora fragment other languages more aggressively. The same sentence in English and in a less-represented language can differ severalfold in token count — meaning real differences in cost, latency and usable context for speakers of those languages.
- Whitespace and formatting cost money. Indentation, repeated punctuation and decorative formatting all consume tokens.
Embeddings: turning tokens into geometry
A token ID is just an index — arbitrary, carrying no meaning. The embedding layer converts each ID into a vector of hundreds or thousands of numbers, and these vectors are learned during training.
What the model learns is a space in which position encodes meaning. Tokens used in similar ways end up near each other. The classic demonstration is that directions in this space correspond to semantic relationships: the offset from “king” to “queen” resembles the offset from “man” to “woman”. Nobody programmed a concept of gender; it falls out of the statistics of usage.
This is why the phrase “the model understands” is contested but not absurd. There is no comprehension in any human sense. There is a high-dimensional geometry in which genuine semantic structure is encoded, well enough to support reliable inference about relationships the model was never explicitly taught.
Why embeddings matter outside the model
Embeddings are also a product in their own right, and the foundation of modern search. Because semantically similar text lands in nearby regions of the space, you can compare meaning rather than keywords.
The mechanism: run each document through an embedding model, store the resulting vectors, then embed an incoming query the same way and find the nearest vectors by cosine similarity. A query about “reducing cloud spend” can surface a document titled “lowering AWS costs” with no shared keywords. This is the retrieval half of retrieval-augmented generation.
Several practical considerations govern this in production:
- Dimensionality is a tradeoff. Common embedding models output between several hundred and a few thousand dimensions. More dimensions capture more nuance and cost more to store and search. Some current models support truncation to smaller sizes with modest quality loss, letting you tune the tradeoff after the fact.
- Vectors are model-specific. Embeddings from different models are not comparable. Changing embedding model means re-embedding your entire corpus.
- Chunking is often the bottleneck. Embedding an entire long document averages its meaning into mush. How you split text before embedding frequently matters more to retrieval quality than which model you chose.
- Hybrid retrieval usually wins. Semantic similarity is weak on exact matches — product codes, proper nouns, precise figures. Production systems generally combine vector search with traditional keyword search, and adoption of hybrid approaches has risen sharply.
The layer worth remembering
Tokens determine what the model can perceive; embeddings determine what it can relate. Most of the time this machinery is invisible and you can reason about models as if they read words. The moments it becomes visible — a miscounted letter, a language that costs triple, a search that misses an obvious match — are worth recognising for what they are, because they are generally not fixable by better prompting.
Related: what is a large language model and transformers explained.
Last reviewed: September 2026.
