Almost every AI system you have used in the last five years runs on an architecture described in a 2017 paper called Attention Is All You Need. Understanding one idea from that paper — attention — explains most of what is distinctive about modern AI, including why it scaled when earlier approaches stalled, and why running it is so expensive.
The problem transformers solved
Before transformers, sequence models read text the way you do: one word at a time, left to right, carrying a compressed summary of everything seen so far. Recurrent networks and LSTMs worked this way, and they had two crippling properties.
First, information degraded over distance. By the time the model reached the end of a long paragraph, the beginning had been squeezed through so many sequential updates that little survived. Second, and more fatally for scaling, the sequential dependency meant the computation could not be parallelised. Word fifty could not be processed until word forty-nine was done. Modern accelerators are enormously parallel machines, and this architecture could not use them.
What attention actually does
Attention discards the sequential bottleneck entirely. Instead of passing a summary forward, every position in the text looks directly at every other position and decides how much each one matters to it.
Concretely, each token produces three vectors, by convention called query, key and value. Think of it as a lookup. The query is what this token is looking for. The key is what each other token advertises about itself. Comparing a query against all the keys produces a set of relevance scores, which are normalised into weights. The output for that position is then the weighted sum of all the values.
The standard illustration: in “the animal didn’t cross the street because it was too tired”, resolving it requires attending to animal rather than street. No rule encodes this. The weights are learned, because getting such relationships right improves next-token prediction.
Crucially, every position does this simultaneously. The whole operation is a few large matrix multiplications — precisely the shape of computation that GPUs excel at. This is the unlock. Transformers did not merely perform better; they performed better and could absorb vastly more compute, which is what made scaling laws exploitable.
The pieces around attention
- Multi-head attention. Rather than one attention operation, the model runs many in parallel, each with its own learned projections. Different heads specialise — some track syntax, some track long-range reference, some appear to do almost nothing. One head alone would force a single notion of relevance.
- Positional encoding. Attention is inherently order-blind; it sees a set, not a sequence. Position information has to be injected deliberately. Modern models typically use rotary position embeddings, which encode relative rather than absolute position and generalise better to longer inputs.
- Feed-forward layers. After each attention step, a position-wise network processes each token independently. These layers hold most of the parameters, and a good deal of evidence suggests they are where factual knowledge is stored.
- Residual connections and normalisation. Each sublayer adds to its input rather than replacing it, creating a path for gradients to flow through dozens of layers. Without this, deep stacks do not train.
Stack that block many times, and you have a transformer. The architecture is remarkably uniform — depth and width change, the block does not.
The quadratic problem
Attention’s strength carries its central cost. If every token attends to every other token, the work grows with the square of the sequence length. Double the input and you quadruple the attention computation. The memory required to hold the intermediate state grows the same way.
This single fact drives much of the engineering in the field. It is why long context was hard to achieve, why it is priced at a premium, and why a great deal of research targets attention specifically. Mitigations in current use include sparse attention patterns, where each token attends to a subset rather than everything; grouped-query attention, which shares key and value projections across heads to shrink the cache; and memory-efficient exact implementations such as FlashAttention, which avoid ever materialising the full attention matrix. Several frontier labs now ship proprietary sparse-attention variants and describe them only in general terms.
The practical consequence for anyone building on these models is covered in context windows explained.
Why the architecture has lasted
Nearly a decade is a long time for one design to remain dominant in a fast-moving field, and plenty of challengers have appeared — state-space models such as Mamba, various linear-attention schemes, hybrid designs. Some are genuinely competitive on particular axes, especially throughput at long sequence lengths.
None has displaced the transformer, for a reason worth naming: the advantage is now partly sociological rather than purely technical. An enormous accumulation of tooling, kernels, hardware design, training know-how and published results assumes transformers. A replacement must be better by a wide enough margin to justify abandoning all of it. The likelier path, visible in current frontier models, is the transformer absorbing good ideas — sparse routing, cheaper attention variants — while keeping its basic shape.
Related reading: what is a large language model and tokens and embeddings.
Last reviewed: September 2026.
