Promythicus FR root@promythicus.com

Promythicus / Blog

An outline the AI writes itself beats searching the contract

By Bardia Shahrestani, Promythicus Inc. · Published · Updated

On the CUAD contract-review test, a local model that first wrote a one-line label for each part of a contract, then chose which parts to read from that outline, quoted 81% of the clauses lawyers had marked. The best search quoted 71% while reading the same amount of text, and reading the whole contract quoted 82%. No training was involved. The outline costs 22 to 31 seconds per contract, once; a question then takes about 2.7 seconds. Everything ran on one workstation with one 16 GB graphics card and no cloud.

Summary of the results: clauses quoted 71% with search, 77% with search and outline, 81% with the outline, 82% reading the whole contract

Where the first test left off

In our first test, searching a contract and reading only the best passages was fast but cost accuracy: the model quoted 71% of the clauses lawyers had marked, against 82% when it read the whole contract. The model quoted nearly everything the search put in front of it, so the loss was in the search.

The idea

A recent paper, STAIR, takes a different route. Instead of matching the question against passages, it shows a language model the document’s table of contents and asks which section holds the answer. A person does the same with a long document.

We could not use the method as published, for two reasons. It trains a model on each collection of documents. And it needs a real table of contents, which most contracts in this test do not have. So we tested the plain version, with no training:

Same data, same machine and same model as the first test: CUAD, one workstation with one 16 GB graphics card, a mixed-precision variant of Qwen 3.8 27B that runs on a single 16 GB GPU. The search it is compared with is the best one from the first test.

Is the right clause in what gets read

On 30 of the 102 test contracts (374 questions that have an answer, 820 marked clauses), we checked what each method would hand the model, at the same reading allowance.

What is readRight clause found, about 6,000 charactersMarked clauses coveredRight clause found, about 12,000 characters
Best search87%70%93%
Outline91%82%94%
Search and outline combined93%81%97%

The outline finds the right place a little more often than the search. The larger difference is in the third column: when a question has several marked clauses, the outline brings in far more of them.

What the model then answered

Then the full run, on the same ten contracts and 410 questions as the first test, with the same scoring.

Way of readingMarked clauses quotedQuotes that are word for wordText read per question
Best search, then read71%98%about 1,200 tokens
Search and outline combined77%95%about 1,200 tokens
Outline, 4 parts81%95%about 1,200 tokens
Outline, 8 parts82%94%about 2,100 tokens
Read the whole contract82%91%about 8,400 tokens

Reading four parts chosen from the outline, about a seventh of the contract, the model quoted 81% of the marked clauses. Reading the whole contract, it quoted 82%.

Combining the two methods did worse than the outline alone, even though it found the right passage slightly more often. The outline reads whole parts of the contract in their order; the combination reads short pieces from many places. Our reading is that the model quotes more completely from continuous text.

What it costs

For a firm, the outline is written when a document is filed, not when someone asks. A thousand contracts would take about nine hours of machine time, once (about six after the changes below).

Making it faster

We then tried to cut those times, on the same ten contracts and questions.

ChangeOutline, per contractPer question, picking and answeringMarked clauses quoted
First run31 s4.3 s81%
The model names 4 parts instead of up to 831 s4.0 s82%
The same, on the server’s faster setting31 s2.6 s82%
The same, with the labels written eight at a time22 s2.7 s82%
A small model (4 billion parameters) writes the labels3 s3.9 s72%

A question went from 4.3 seconds to 2.7 with the same accuracy, and the outline from 31 seconds per contract to 22. On the faster setting the model drafts two words ahead and checks them, at the price of a shorter maximum document. Writing the labels eight at a time gave less than we expected, because one request already uses most of the graphics card.

The small model wrote the outline ten times faster, but its labels were vaguer, the right part was read less often (88% of questions against 95%), and the result fell back to what the search gave. The outline is worth the larger model’s time.

Limits

Questions and answers

What is an outline of a contract, here?
The contract is cut into parts of 1,500 characters and the model writes one line per part naming the topics it covers. For each question the model reads those lines and names the parts to read.
How much better was it than search?
Reading the same amount of text, the model quoted 81% of the clauses lawyers had marked with the outline, against 71% with the best search we had found. Reading the whole contract gave 82%.
Does it need training?
No. The STAIR paper that suggested the idea trains a model; we used the same general model for the labels, the choice of parts and the answer, with no training.
What does it cost?
In the first run, 31 seconds of machine time per contract, once, to write the outline, and 4.3 seconds per question. After tuning, 22 seconds per contract and 2.7 seconds per question, with the same accuracy.
Did any of this use a cloud service?
No. Everything ran on one workstation with one 16 GB graphics card.

Sources