Making a local AI answer from your own documents, with the page shown
There are two ways. For a large library, the system searches for the passages that match the question and gives only those to the model (this is called retrieval, or RAG). For a small set of documents that all matter, such as the leases in one file, it has the model read every document in turn. Either way, the model should return the exact sentence it relied on, so the screen can open that page.
Why a chatbot alone is not enough
A model on its own answers from what it learned in training. It knows nothing about your files. To answer from them, the text of the right documents has to be placed in front of it with the question. The work is in choosing which text, and in proving where the answer came from.
Way one: search, then read (RAG)
- Every document is split into passages, and each passage is turned into a list of numbers that stands for its meaning (an embedding). This is done once, on the same machine.
- When a question comes in, the system finds the passages closest in meaning, usually combined with ordinary keyword search.
- Only those passages go to the model with the question.
Good for: thousands of documents, questions whose answer sits in one or two places.
Weak at: questions about the whole set. “Which of these contracts have no renewal clause?” cannot be answered by finding matching passages, because the answer is in what is missing.
Way two: read every document
For one matter or one client file, the set is small enough to read in full. The model reads each document and returns one entry per document: the relevant fact, whether the question applies, and the sentence that shows it.
Good for: “all of them” questions, tables across a set, deadlines.
Cost: time grows with the number of documents. Reading each document once and keeping its key terms cuts most of it.
Showing the source
An answer nobody can check is not usable in a professional file. What works:
- Ask the model to return the sentence it relied on, word for word, along with its answer.
- Have the program find that sentence in the document’s text. If it is not there, treat the answer as unsupported.
- Open the page and mark the sentence, so the reader checks it in two seconds.
Getting the text out
Typed PDFs give up their text directly. Scans need character recognition first, and its mistakes carry through, so check a sample. Tables and handwritten notes are the usual trouble spots.
Keeping it all local
Every piece of this runs on the server: text extraction, the search index, the embedding model and the main model. Nothing in the method needs an outside service. If a product’s version of it calls one, the documents are leaving the building.
Questions and answers
- What is RAG?
- Retrieval-augmented generation: the system searches your documents for the passages that match a question and gives those passages to the AI model to answer from.
- Can a local AI show where its answer came from?
- Yes. The model returns the sentence it relied on, the program finds that sentence in the document, and the screen opens the page with it marked.
- Does a local AI need to be trained on our documents?
- No. The documents are placed in front of the model when a question is asked. The model itself is not changed.