Language Models are Few-Shot Learners (GPT-3)
Showed that a big enough language model can pick up a new task from a few examples in its prompt, with no retraining.
Train a 175-billion-parameter language model on 300 billion tokens, then give it tasks only through the prompt: an instruction plus zero, one or a few examples, with no gradient updates. Few-shot performance kept rising with scale.
Fine-tuning needed a labelled dataset of thousands of examples for every new task, unlike people, who can learn a task from a few.
Eight model sizes from 125M to 175B parameters, all trained on 300B tokens, then tested zero-shot, one-shot and few-shot inside a 2,048-token context window.
Few-shot accuracy on LAMBADA reached 86.4%, up from a prior best of 68.0%. TriviaQA rose from 64.3% zero-shot to 71.2% few-shot.
Why it matters
GPT-3 made prompting the main way to use language models and made scale the field’s central bet. Its API, and the instruction-tuned models that followed, led to ChatGPT.
This is a lite card. The full version adds the core idea in technical detail, limitations and open questions, what it builds on and led to, and key terms.