The context window explained: how much can a local AI read at once?
The context window is the amount of text a model can hold in view while it answers: the question, the documents it was given, and its own reply. It is counted in tokens, pieces of words. On our 16 GB card the model holds about 113,000 tokens, roughly 250 pages. Anything beyond the window has to be read in turns or found by search first.
Tokens
Models do not read letters or words. They read tokens, which are short pieces of text. In English a token is about three quarters of a word. French takes somewhat more tokens for the same content. A full page of a contract is around 450 tokens.
What takes up the window
- The instructions the system gives the model.
- The documents or passages placed in front of it.
- The conversation so far.
- The answer it is writing.
When the total would pass the limit, something has to be left out. A model has no memory outside this window: what is not in it does not exist for that answer.
Why not make it as long as possible
- Memory. Every token in view takes graphics memory. On our machine, about 23 MB per 1,000 tokens.
- Speed. Reading a long context takes time before the first word of the answer, and writing slows as the window fills: from about 41 tokens a second to about 28 on our card.
- Attention. Models find a fact more reliably at the start or end of a long context than in the middle. A full window is not read as carefully as a short one.
When the documents are bigger than the window
- Read in turns. Each document is read by itself and its key facts are kept. The question is then answered from the facts. This suits a set of contracts.
- Search first. Find the passages that match the question and give the model only those. This suits a large library.
- Both. Search to pick the documents, then read each picked document in full.
See making a local AI answer from your own documents for how these work.
Rough sizes
| Text | Tokens, about |
|---|---|
| One page | 450 |
| A commercial lease, 40 pages | 18,000 |
| A 250-page file | 113,000 |
| A filing cabinet | Millions. Needs search |
Questions and answers
- What is a context window?
- The amount of text an AI model can hold in view while answering: the question, the documents given to it and its reply, counted in tokens.
- How many pages can a local AI model read at once?
- On a 16 GB graphics card we fit about 113,000 tokens, roughly 250 pages. Larger sets are read in turns or narrowed by search first.
- What is a token?
- A short piece of text that a model reads as one unit. In English it is about three quarters of a word.