A large language model is a system that has learned the statistical structure of human language well enough to continue any passage of text plausibly. That is the entire mechanism. Everything else — the chat interface, the reasoning, the tool use, the agents — is built on top of that one capability.
Next-token prediction is the whole trick
During training, a model is shown an enormous quantity of text with the next word hidden, and asked to guess it. It guesses, the guess is compared to the actual word, and the model’s internal parameters are nudged to make the correct answer slightly more likely next time. Repeat this across trillions of words.
What makes this more interesting than autocomplete is that predicting text well requires learning the things the text is about. To finish the sentence “the capital of France is” you need a fact. To finish “the patient’s symptoms suggest” you need medical associations. To finish a half-written function you need to model program semantics. Compression of language turns out to require a working model of much of what language describes.
This also explains the characteristic failure mode. The model is optimising for plausible continuation, not for truth. A confident, fluent, wrong answer is exactly what you would expect from a system trained this way — see our explainer on why AI models hallucinate.
Training happens in distinct stages
A shipped model is the product of several different training processes, and conflating them causes most public confusion about what models “know” or “want”.
Pretraining
The next-token prediction phase described above, run over a very large text corpus. This is where almost all knowledge and capability is acquired, and where almost all of the compute is spent. The result is a model that can continue text but has no particular inclination to be helpful, harmless, or conversational.
Post-training
A much smaller, much more deliberate phase that shapes behaviour rather than knowledge. Supervised fine-tuning on example conversations teaches the model the assistant format. Reinforcement learning from human feedback, and related methods, then tune it toward responses people rate highly. Anthropic’s published Constitutional AI approach is a variant in which the model evaluates its own outputs against written principles to generate that preference data.
Post-training is why two models trained on similar data can have very different personalities, refusal behaviour, and verbosity. It is a thin layer, and it is the layer vendors compete on hardest.
Reasoning training
The newest stage. Models are trained with reinforcement learning on tasks where an answer can be automatically checked — mathematics, code that either passes tests or does not — which rewards working through a problem step by step before answering. OpenAI states plainly that its reasoning models are trained to reason through reinforcement learning. DeepSeek’s R1 release remains the most fully documented public recipe for this. See reasoning models explained.
What “large” actually means in 2026
Parameter count was once the headline number. It has become a much less useful one, for two reasons.
First, the frontier has moved to sparse mixture-of-experts architectures, where only a fraction of the network activates for any given token. Among models that publish their configurations, active parameters commonly run at roughly two to four per cent of the total — a model may hold well over a trillion parameters while using only tens of billions per token. A single “size” figure no longer describes the compute cost of running it.
Second, no major closed frontier lab publishes parameter counts at all. Any specific figure quoted for a leading proprietary model is inference or rumour. Treat it accordingly.
What is publicly comparable: context window, price per million tokens, and measured performance on tasks. Those are the numbers worth tracking.
What follows from the architecture
Several widely-noted model behaviours are direct consequences of how these systems work, not bugs to be patched:
- No inherent access to current information. A model’s knowledge is fixed at its training cutoff. Anything later has to arrive through retrieval or tools.
- No memory between conversations by default. What looks like memory is the conversation being re-sent each turn, or an explicit memory system built around the model.
- Arithmetic and exact recall are structurally awkward. The model predicts token sequences; it does not execute operations. This is why tool use matters so much.
- Uncertainty is poorly expressed. Nothing in the training objective rewards saying “I don’t know”, and most benchmarks actively punish it.
The terms you will keep meeting
- Token — the unit of text a model reads and writes, typically a word fragment. See tokens and embeddings.
- Parameter — a single learned weight. Collectively, what the model “knows”.
- Context window — how much text the model can consider at once. See context windows explained.
- Inference — running a trained model to produce output, as opposed to training it.
- Transformer — the neural network architecture underneath essentially every current LLM. See transformers explained.
The useful mental model
An LLM is best thought of as a very well-read, very fast, slightly unreliable reasoning engine with no memory and no access to the world except what you hand it. Nearly every serious application of these systems is an exercise in compensating for one of those four limitations — retrieval for knowledge, memory systems for persistence, tools for action, and verification for reliability.
Last reviewed: September 2026. Architectural norms and published model specifications change frequently; figures here reflect vendor documentation at time of review.
