Promythicus FR root@promythicus.com

Promythicus / Blog

What hardware does a private AI server need?

By Bardia Shahrestani, Promythicus Inc. · Published

The graphics card’s memory is what matters. The model file has to fit in it, with room left over for the text being read. A 16 GB card runs a 27-billion-parameter model at 4-bit precision, which is enough for accurate work on documents. Everything else in the machine is ordinary.

The rule of thumb

Memory needed on the card = the size of the model file + memory for the context (the text the model holds in view).

What we measured

Our demo runs on one workstation with a single 16 GB graphics card and a 27-billion-parameter open-weight model in a 14.3 GB file, served by llama.cpp:

Value
Memory used with the model loadedAbout 13.2 GB, before any context
Memory per 1,000 tokens of contextAbout 23 MB
Longest context that fitsAbout 113,000 tokens, roughly 250 pages
Writing speedAbout 41 tokens a second, about 28 with the context full
With speculative drafting turned onAbout 65 tokens a second, with a shorter context of about 62,000 tokens
What happens past the card’s limitThe runner stops mid-answer. Leave a margin

A token is about three quarters of a word. Forty tokens a second is faster than anyone reads.

Sizing by office

The rest of the machine

Can it run without a graphics card?

Yes, slowly. On a processor alone, a model of useful size writes a few tokens a second, and reading a long document first can take minutes. Computers with large shared memory, such as recent Apple desktops, sit in between and are a reasonable choice for a small office.

Questions and answers

How much graphics memory does a private AI server need?
16 GB is a practical minimum for document work: it runs a 27-billion-parameter model at 4-bit precision. 24 GB or more gives room for longer documents and more than one user at a time.
How fast is a local AI model?
On our 16 GB card a 27-billion-parameter model writes about 41 tokens a second, around 30 words a second, which is faster than reading speed.
How many pages can a local model read at once?
On a 16 GB card we fit a context of about 113,000 tokens, roughly 250 pages. Larger sets of documents are read in turns.

Sources