Promythicus FR root@promythicus.com

Promythicus / Blog

Can several people use a private AI server at the same time?

By Bardia Shahrestani, Promythicus Inc. · Published

Yes. The model runner answers several questions side by side on one graphics card. Each question in progress takes a share of the card’s memory for the text it is reading, and each answer comes a little slower when the card is busy. A small office is well served by one card; a large one needs more memory or a second card.

How it works

The model is loaded once. The runner then keeps several slots, each holding one conversation. With llama.cpp:

llama-server -m model.gguf -c 65536 -np 4 -ngl 99

-np 4 opens four slots. The context given by -c is shared among them, so this example gives each person about 16,000 tokens, roughly 35 pages in view at a time.

The trade

When all slots are busy, the next question waits in line. For a question that takes six seconds, the wait is rarely noticed.

What staff notice

With the card answering one person, text appears faster than anyone reads. With several at once, the card’s total output goes up and each person’s share goes down. In an office, people rarely ask in the same few seconds, so most questions get the whole card.

Sizing

OfficeSetup
Up to about five peopleOne 16 or 24 GB card, two to four slots
Ten to twenty people24 GB or more, four to eight slots, or two cards
Long documents for everyoneMore memory rather than more slots: 48 GB or two cards

These are starting points. The honest way to size it is to count how many questions your office asks in its busiest ten minutes during a trial.

Keeping people’s work apart

Slots share the model, not the conversations. One person’s documents and questions are never placed in another person’s slot. Who may ask about which files is decided by accounts and permissions, outside the model: see how to secure a private AI server.

Heavy jobs

Reading a whole archive, or character recognition on thousands of scans, can fill the card for hours. Run those at night so the daytime slots stay free.

Questions and answers

How many people can use one private AI server?
A single 16 or 24 GB graphics card serves an office of about five comfortably. Larger offices need more graphics memory or a second card.
Does a private AI get slower with more users?
Each answer is somewhat slower when several questions run at once, because they share the card. Since people rarely ask at the same moment, most questions run at full speed.
Can users see each other’s questions on a shared AI server?
No. Each conversation is kept in its own slot, and access to documents is controlled by each person’s account.

Sources