How to choose an open-weight AI model for private use
Choose the largest model whose 4-bit file fits your graphics card with room for context, whose licence allows commercial use, and that reads your languages well. Then test two or three candidates on your own documents with questions you already know the answers to. The test decides, not the public rankings.
What “open-weight” means
The company that trained the model has published the file. You download it once and run it on your own hardware. Nothing goes back to them. Several large companies and labs publish such models, and new ones arrive every few months.
1. The licence
Each model has its own. Some are fully permissive. Some limit commercial use, set conditions above a certain company size, or carry a use policy. Read it before building on a model. This is the step most often skipped.
2. Size against memory
Bigger models read more carefully and follow instructions better. The limit is the card: see what hardware a private AI server needs. On a 16 GB card, models around 24 to 32 billion parameters at 4-bit precision are the sweet spot.
3. Precision (quantization)
Model files come in several precisions. Lower precision makes the file smaller and slightly less exact.
- 4-bit is the usual choice: little measurable loss for most work at about a third of the full size.
- Below 4-bit the loss becomes noticeable. A smaller model at 4-bit usually beats a larger one at 2 or 3.
- 5, 6 or 8-bit are worth it only if you have memory to spare.
4. Languages
For an office in Quebec the model has to read and write French as well as English, including legal and accounting vocabulary. Models differ a lot here. Test it directly: ask the same question of the same document in both languages.
5. Context length
Check how much text the model can hold at once, and how much of that your card’s memory lets you use. A long lease is 15,000 to 30,000 tokens.
6. Structured answers
For tables and extraction, the model must return a fixed structure reliably. Runners such as llama.cpp can force the output to follow a structure, which removes most of the difference between models on this point.
7. Test on your own documents
- Take ten real documents, with names removed if needed.
- Write fifteen to twenty questions and the correct answers, by hand.
- Run each candidate model through all of them, with the same settings.
- Count right answers, and read the wrong ones. A model that is wrong in a way you can catch is better than one that is wrong confidently.
Keep the test. When a new model comes out, an hour tells you whether to switch.
Questions and answers
- What is an open-weight AI model?
- A model whose trained file is published, so anyone can download it and run it on their own hardware without sending data to its maker.
- Can open-weight models be used commercially?
- Many can, but each has its own licence. Some restrict commercial use or set conditions, so read the licence first.
- Does 4-bit quantization hurt accuracy?
- Slightly. For most document work the loss is small, and a larger model at 4-bit usually does better than a smaller one at full precision in the same memory.