Promythicus FR root@promythicus.com

Promythicus / Blog

What we learned making a local model accurate on legal documents

By Bardia Shahrestani, Promythicus Inc. · Published

A local model read our test leases correctly almost every time, and failed in two specific places: comparing dates, and leaving a document out of a list. Four changes fixed it: one structured entry per document, a required quoted sentence for every fact, date arithmetic done by the program, and no randomness. After them it answered 16 of our 16 test questions correctly in about six seconds each.

These are notes from building our own assistant, on five sample leases and a set of sixteen questions with known answers. The numbers are from that small test and are not a general benchmark.

Where it went wrong

What fixed it

  1. One entry per document. The model must return a fixed structure with one entry for every document: the fact, whether the question applies, the supporting sentence, the page. It cannot skip a document, because an empty entry is visible. The runner enforces the structure, so the output always parses.
  2. A quoted sentence for every fact. The program looks for that sentence in the document. A fact with no matching sentence is flagged.
  3. Arithmetic in code. The model extracts the dates. The program counts the days. Models are good readers and poor calculators.
  4. No randomness. The setting that makes a model’s wording vary (temperature) is at zero for extraction, so the same question gives the same answer.

With those in place the fast mode was right on all sixteen questions, at about six seconds each, and the slow thinking mode was no longer needed.

What we would tell anyone building one

Questions and answers

Are local AI models accurate enough for legal documents?
For reading and extracting from documents, yes, when the system around the model is built for it. In our test a local model answered 16 of 16 questions on five leases correctly once dates were computed by the program and every fact had to carry its source sentence.
Why do AI models make mistakes with dates?
They read dates correctly but are unreliable at arithmetic, such as counting the days between two dates. The fix is to have the model extract the dates and the program do the counting.
How do you stop an AI from skipping a document?
Require one structured entry per document in its answer. A missing or empty entry is then visible and can be checked.

Sources