What we learned making a local model accurate on legal documents
A local model read our test leases correctly almost every time, and failed in two specific places: comparing dates, and leaving a document out of a list. Four changes fixed it: one structured entry per document, a required quoted sentence for every fact, date arithmetic done by the program, and no randomness. After them it answered 16 of our 16 test questions correctly in about six seconds each.
These are notes from building our own assistant, on five sample leases and a set of sixteen questions with known answers. The numbers are from that small test and are not a general benchmark.
Where it went wrong
- Dates. Asked which leases had a notice deadline in the next ninety days, the model in its fast mode dropped one that qualified. It can read a date. It is unreliable at counting days between two of them.
- Lists. Asked a question about all five documents in free text, it sometimes answered about four.
- Thinking mode. Letting the model reason step by step before answering fixed the date mistake. It also took the answer from about 6 seconds to between 12 and 20.
What fixed it
- One entry per document. The model must return a fixed structure with one entry for every document: the fact, whether the question applies, the supporting sentence, the page. It cannot skip a document, because an empty entry is visible. The runner enforces the structure, so the output always parses.
- A quoted sentence for every fact. The program looks for that sentence in the document. A fact with no matching sentence is flagged.
- Arithmetic in code. The model extracts the dates. The program counts the days. Models are good readers and poor calculators.
- No randomness. The setting that makes a model’s wording vary (temperature) is at zero for extraction, so the same question gives the same answer.
With those in place the fast mode was right on all sixteen questions, at about six seconds each, and the slow thinking mode was no longer needed.
What we would tell anyone building one
- Write the test questions before choosing a model, with answers you have checked by hand. Run them again after every change.
- Test on documents like yours. A model that scores well in public tables can still misread your kind of clause.
- Decide what the model does and what the program does. Reading and summarizing go to the model. Counting, sorting, comparing and totalling go to the program.
- Show the source every time. People forgive a slow answer. They stop using a tool after one wrong answer they could not check.
Questions and answers
- Are local AI models accurate enough for legal documents?
- For reading and extracting from documents, yes, when the system around the model is built for it. In our test a local model answered 16 of 16 questions on five leases correctly once dates were computed by the program and every fact had to carry its source sentence.
- Why do AI models make mistakes with dates?
- They read dates correctly but are unreliable at arithmetic, such as counting the days between two dates. The fix is to have the model extract the dates and the program do the counting.
- How do you stop an AI from skipping a document?
- Require one structured entry per document in its answer. A missing or empty entry is then visible and can be checked.