What hardware does a private AI server need?
The graphics card’s memory is what matters. The model file has to fit in it, with room left over for the text being read. A 16 GB card runs a 27-billion-parameter model at 4-bit precision, which is enough for accurate work on documents. Everything else in the machine is ordinary.
The rule of thumb
Memory needed on the card = the size of the model file + memory for the context (the text the model holds in view).
- Model file. At 4-bit precision, the usual choice, a model takes roughly 0.5 to 0.6 GB per billion parameters. A 27-billion-parameter model is a file of about 14 GB.
- Context. Grows with how much text you let the model read at once. On our machine it costs about 23 to 29 MB per 1,000 tokens.
What we measured
Our demo runs on one workstation with a single 16 GB graphics card and a 27-billion-parameter open-weight model in a 14.3 GB file, served by llama.cpp:
| Value | |
|---|---|
| Memory used with the model loaded | About 13.2 GB, before any context |
| Memory per 1,000 tokens of context | About 23 MB |
| Longest context that fits | About 113,000 tokens, roughly 250 pages |
| Writing speed | About 41 tokens a second, about 28 with the context full |
| With speculative drafting turned on | About 65 tokens a second, with a shorter context of about 62,000 tokens |
| What happens past the card’s limit | The runner stops mid-answer. Leave a margin |
A token is about three quarters of a word. Forty tokens a second is faster than anyone reads.
Sizing by office
- One to five people, one question at a time: one 16 GB or 24 GB card.
- A busy office, several questions at once: a 24 GB card or larger, or two cards. Each question in progress needs its own share of context memory.
- Very long documents or larger models: 48 GB or more, on one workstation-class card or split across two.
The rest of the machine
- System memory: 32 GB or more.
- Drive: a fast solid-state drive, 1 TB or more. Model files are 10 to 40 GB each and you will keep several.
- Processor: any recent one. It matters little once the model is on the card.
- Power and noise: a card under load draws a few hundred watts and its fans are audible. Put the machine in a closet or server room, not beside a desk.
Can it run without a graphics card?
Yes, slowly. On a processor alone, a model of useful size writes a few tokens a second, and reading a long document first can take minutes. Computers with large shared memory, such as recent Apple desktops, sit in between and are a reasonable choice for a small office.
Questions and answers
- How much graphics memory does a private AI server need?
- 16 GB is a practical minimum for document work: it runs a 27-billion-parameter model at 4-bit precision. 24 GB or more gives room for longer documents and more than one user at a time.
- How fast is a local AI model?
- On our 16 GB card a 27-billion-parameter model writes about 41 tokens a second, around 30 words a second, which is faster than reading speed.
- How many pages can a local model read at once?
- On a 16 GB card we fit a context of about 113,000 tokens, roughly 250 pages. Larger sets of documents are read in turns.