Promythicus FR root@promythicus.com

Promythicus / Blog

How to set up your own private AI server, step by step

By Bardia Shahrestani, Promythicus Inc. · Published

To set up a private AI server you need one computer with a graphics card, a model runner (llama.cpp or Ollama), an open-weight model file, and a chat screen your staff can open in a browser. Install them in that order, then block the machine’s outgoing internet access. A technical person can have a basic one answering questions in an afternoon.

1. The machine

Any desktop or small server with a recent NVIDIA graphics card works. The card’s memory decides which models you can run: 16 GB is a practical minimum for useful work on documents, 24 GB or more is comfortable. Add 32 GB or more of system memory and a fast drive. See what hardware a private AI server needs for sizing.

Install Ubuntu Server or another Linux. Turn on full-disk encryption during the install: the machine will hold client files.

2. The graphics driver

Install the NVIDIA driver and check that the card is seen:

sudo ubuntu-drivers install
sudo reboot
nvidia-smi

nvidia-smi should list the card and its memory.

3. The model runner

The runner is the program that loads the model and answers requests. Two common choices:

With Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull <model name>
ollama run <model name>

With llama.cpp, after building it and downloading a model file in GGUF format:

llama-server -m model.gguf -c 32768 -ngl 99 --host 127.0.0.1 --port 8080

-c is the context length in tokens (how much text the model can read at once) and -ngl 99 puts every layer on the graphics card. Both runners answer on the same kind of web address that cloud AI services use, so most software written for those services can point at your machine instead.

4. The model

Pick an open-weight model whose file fits in your card’s memory with room left for the text it reads. Check its licence allows commercial use. Then test it on your own documents: public scores say little about how a model reads a lease or a tax slip. See how to choose an open-weight model.

5. A chat screen for the office

Open WebUI gives staff a familiar chat page, with accounts. It runs on the same machine:

docker run -d -p 3000:8080 -v open-webui:/app/backend/data \
  --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Staff then open http://<server address>:3000 from their own desks. Point it at your runner in its settings.

6. Cut it off from the internet

Do this after everything is downloaded. Allow the office network in, and nothing out:

sudo ufw default deny incoming
sudo ufw default deny outgoing
sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp
sudo ufw allow out to 192.168.1.0/24
sudo ufw enable

Replace 192.168.1.0/24 with your office network. Then check it: see how to verify that an AI is really private.

7. What this gives you, and what it does not

You now have a private chatbot. It answers from what the model already knows and from text you paste in. It does not yet answer from your files, show the page an answer came from, or build tables across a set of documents. That is the larger part of the work: see making a local AI answer from your own documents.

If you would rather not build it yourself, this is what we do: we build, install and maintain private AI servers for professional firms across Canada.

Questions and answers

Can I build a private AI server myself?
Yes. With one computer, a graphics card with 16 GB of memory or more, and free software such as llama.cpp or Ollama and Open WebUI, a technical person can have a private chatbot running in an afternoon.
What software do I need for a private AI server?
A model runner (llama.cpp or Ollama), an open-weight model file, and a chat interface such as Open WebUI. All three are free.
Does a private AI server need a subscription?
No. Open-weight models and the software that runs them are free to use; you pay for the hardware and the electricity.

Sources