Attention Is All You Need
Introduced the Transformer, a model built only from attention. It dropped recurrence entirely, trained far faster in parallel, and became the base of almost every modern language model.
Replace recurrent and convolutional layers with stacked self-attention. Every token looks at every other token at once, so training runs in parallel across the whole sequence and long-range links stop fading.
Recurrent sequence models process tokens one at a time. That blocks parallel training and makes distant words hard to connect.
An encoder-decoder built from multi-head self-attention and feed-forward layers, with positional encodings to keep word order.
New state of the art on WMT 2014 translation: 28.4 BLEU English to German and 41.8 English to French, at a fraction of earlier training cost.
What is new vs prior work
Attention already existed as an add-on to recurrent translation models. This paper’s claim is that attention alone is enough. Removing recurrence makes the path between any two positions a single step, and lets the whole sequence train in parallel on GPUs. The big model trained in 3.5 days on eight GPUs.
The core idea, technically
Optional · grad levelEach token is projected into a query, a key and a value. Query-key dot products, scaled by √dₖ to keep the softmax gradients stable, decide how much of each value flows into the output. Multi-head attention runs several of these in parallel on smaller projections, so the model can track several kinds of relationship at once.
Limitations and open questions
- Attention cost grows with the square of sequence length, which led to years of work on efficient and long-context attention.
- The paper tested translation and English constituency parsing only. The move into vision, speech and biology came from later papers.
- It used fixed sinusoidal positions. Later models moved to learned, relative and rotary position encodings.