Promythicus FR root@promythicus.com

Promythicus / Blog

The context window explained: how much can a local AI read at once?

By Bardia Shahrestani, Promythicus Inc. · Published

The context window is the amount of text a model can hold in view while it answers: the question, the documents it was given, and its own reply. It is counted in tokens, pieces of words. On our 16 GB card the model holds about 113,000 tokens, roughly 250 pages. Anything beyond the window has to be read in turns or found by search first.

Tokens

Models do not read letters or words. They read tokens, which are short pieces of text. In English a token is about three quarters of a word. French takes somewhat more tokens for the same content. A full page of a contract is around 450 tokens.

What takes up the window

When the total would pass the limit, something has to be left out. A model has no memory outside this window: what is not in it does not exist for that answer.

Why not make it as long as possible

When the documents are bigger than the window

  1. Read in turns. Each document is read by itself and its key facts are kept. The question is then answered from the facts. This suits a set of contracts.
  2. Search first. Find the passages that match the question and give the model only those. This suits a large library.
  3. Both. Search to pick the documents, then read each picked document in full.

See making a local AI answer from your own documents for how these work.

Rough sizes

TextTokens, about
One page450
A commercial lease, 40 pages18,000
A 250-page file113,000
A filing cabinetMillions. Needs search

Questions and answers

What is a context window?
The amount of text an AI model can hold in view while answering: the question, the documents given to it and its reply, counted in tokens.
How many pages can a local AI model read at once?
On a 16 GB graphics card we fit about 113,000 tokens, roughly 250 pages. Larger sets are read in turns or narrowed by search first.
What is a token?
A short piece of text that a model reads as one unit. In English it is about three quarters of a word.

Sources