The context window is the amount of text a model can consider at one time, measured in tokens. It is the most consequential practical limit in applied AI, and the one most often misunderstood — in both directions. People assume the model remembers more than it does, then assume a larger window automatically solves the problem.
The window is the entire working memory
A language model has no persistent state between calls. When you send a message, the model receives the system instructions, the full prior conversation, any retrieved documents, any tool definitions and results, and your new message — all of it, every single turn. It reads that text, produces a response, and retains nothing.
This explains a behaviour that otherwise looks like malfunction. When a long chat “forgets” something you said earlier, the information has not decayed. It has been pushed out of the window to make room, or it is still present but the model is failing to use it. Those are different problems with different fixes, and distinguishing them is most of the diagnostic work.
Note also what shares the space. Tool definitions, retrieved documents and reasoning tokens all consume the same budget as your actual content. An agent loaded with fifty tool schemas may have spent a substantial share of its window before reading a word of your request.
Where window sizes stand
The trajectory has been steep. Early production models worked with a few thousand tokens. As of September 2026, roughly one million tokens is the standard ceiling across leading models from the major labs — enough for a long novel or a substantial code repository. What was a headline differentiator three years ago is now a baseline expectation.
Two qualifications matter. Claims of windows dramatically beyond this circulate widely online and generally do not survive checking against vendor documentation. And output limits are separate and much smaller than input limits — a model that accepts a million tokens may be capped at a small fraction of that in a single response. Maximum output length has quietly become its own axis of competition.
Why bigger is not simply better
Cost scales with what you send
You pay per input token, every call. Filling a million-token window in a multi-turn conversation is expensive, and several vendors apply a long-context surcharge above a threshold — often around double the base rate. The economical pattern is to send what is needed, not everything available. Prompt caching, which discounts repeated prefixes steeply, changes this calculus substantially for applications with stable system prompts.
Attention cost grows quadratically
As covered in transformers explained, attention compares every token with every other token, so the underlying work rises with the square of the length. This shows up as latency, and it is why long-context requests feel sluggish.
Effective use degrades before the limit
This is the most important and least appreciated point. A model’s advertised window and the window over which it reliably reasons are not the same. Retrieval benchmarks have repeatedly shown degradation well before the stated maximum, with a characteristic pattern: material at the very beginning and very end of a long input is used more reliably than material buried in the middle.
The practical implication is to treat position as a design variable. Put instructions and the most critical material at the edges of a long prompt, not the middle. And do not assume that because something is in the window, it has been read.
Managing a finite window
- Retrieval. Rather than loading an entire corpus, fetch the relevant fragments per query. Still the dominant pattern for large knowledge bases — see RAG explained.
- Compaction. When a conversation approaches the limit, summarise the older portion and replace it with the summary. Lossy by construction; what gets dropped is the hard part.
- External memory. Write durable facts to files or a database the model can read on demand, so they survive compaction. This is how multi-session agents maintain state.
- Context editing. Actively remove stale tool results and superseded material rather than letting them accumulate.
- Keeping output out of context. Have tools process large results in a sandbox and return only conclusions, rather than dumping raw output into the window.
The shift in how to think about it
When windows were small, context management was rationing. Now that a million tokens is ordinary, the discipline has inverted: the question is no longer how to fit everything in, but what deserves to be there. Filling the window is easy, costly, and frequently counterproductive. Curating it is the actual skill — and it is now a recognised engineering specialism rather than a prompt-writing trick.
Last reviewed: September 2026. Window sizes and pricing move quickly; figures reflect vendor documentation at time of review.
