BERT: Pre-training of Deep Bidirectional Transformers
Showed that one Transformer encoder, pre-trained on plain text by filling in hidden words, could be fine-tuned to beat specialised models across language tasks.
Pre-train one deep Transformer encoder on unlabelled text by hiding words and predicting them from both sides, then fine-tune it with a single extra layer for each task. It set new records on eleven language tasks.
Earlier pre-trained language models read text in one direction, so each word only saw part of its context.
Masked language modelling (15% of tokens hidden and predicted) plus next sentence prediction, trained on BooksCorpus and English Wikipedia.
GLUE score of 80.5, up 7.7 points. SQuAD v1.1 F1 of 93.2 and SQuAD v2.0 F1 of 83.1.
Why it matters
BERT made “pre-train once, fine-tune for each task” the standard way to build language systems. It also split the field into encoder models like BERT, used to understand text, and decoder models like GPT, used to generate it. Many of the text embedding models behind today’s search and RAG systems still use BERT-style encoders.
This is a lite card. The full version adds the core idea in technical detail, limitations and open questions, what it builds on and led to, and key terms.