An outline the AI writes itself beats searching the contract
On the CUAD contract-review test, a local model that first wrote a one-line label for each part of a contract, then chose which parts to read from that outline, quoted 81% of the clauses lawyers had marked. The best search quoted 71% while reading the same amount of text, and reading the whole contract quoted 82%. No training was involved. The outline costs 22 to 31 seconds per contract, once; a question then takes about 2.7 seconds. Everything ran on one workstation with one 16 GB graphics card and no cloud.
Where the first test left off
In our first test, searching a contract and reading only the best passages was fast but cost accuracy: the model quoted 71% of the clauses lawyers had marked, against 82% when it read the whole contract. The model quoted nearly everything the search put in front of it, so the loss was in the search.
The idea
A recent paper, STAIR, takes a different route. Instead of matching the question against passages, it shows a language model the document’s table of contents and asks which section holds the answer. A person does the same with a long document.
We could not use the method as published, for two reasons. It trains a model on each collection of documents. And it needs a real table of contents, which most contracts in this test do not have. So we tested the plain version, with no training:
- When a contract comes in, it is cut into parts of 1,500 characters and the model writes one line for each part, naming the topics it covers. Those lines are the outline. A typical line: “Development license; development fees; term; renewal; termination; notice.”
- For a question, the model is shown the outline and names the parts most likely to hold the answer. Only those parts are read.
Same data, same machine and same model as the first test: CUAD, one workstation with one 16 GB graphics card, a mixed-precision variant of Qwen 3.8 27B that runs on a single 16 GB GPU. The search it is compared with is the best one from the first test.
Is the right clause in what gets read
On 30 of the 102 test contracts (374 questions that have an answer, 820 marked clauses), we checked what each method would hand the model, at the same reading allowance.
| What is read | Right clause found, about 6,000 characters | Marked clauses covered | Right clause found, about 12,000 characters |
|---|---|---|---|
| Best search | 87% | 70% | 93% |
| Outline | 91% | 82% | 94% |
| Search and outline combined | 93% | 81% | 97% |
The outline finds the right place a little more often than the search. The larger difference is in the third column: when a question has several marked clauses, the outline brings in far more of them.
What the model then answered
Then the full run, on the same ten contracts and 410 questions as the first test, with the same scoring.
| Way of reading | Marked clauses quoted | Quotes that are word for word | Text read per question |
|---|---|---|---|
| Best search, then read | 71% | 98% | about 1,200 tokens |
| Search and outline combined | 77% | 95% | about 1,200 tokens |
| Outline, 4 parts | 81% | 95% | about 1,200 tokens |
| Outline, 8 parts | 82% | 94% | about 2,100 tokens |
| Read the whole contract | 82% | 91% | about 8,400 tokens |
Reading four parts chosen from the outline, about a seventh of the contract, the model quoted 81% of the marked clauses. Reading the whole contract, it quoted 82%.
Combining the two methods did worse than the outline alone, even though it found the right passage slightly more often. The outline reads whole parts of the contract in their order; the combination reads short pieces from many places. Our reading is that the model quotes more completely from continuous text.
What it costs
- Once per contract: writing the outline took 31 seconds per contract on average (324 parts in 308 seconds). The search index took a quarter of a second per contract.
- Per question: the model takes 1.3 seconds to pick the parts, then about 3 seconds to answer. With the search, picking takes 0.16 seconds. The first question on a whole contract took 8 seconds on average.
For a firm, the outline is written when a document is filed, not when someone asks. A thousand contracts would take about nine hours of machine time, once (about six after the changes below).
Making it faster
We then tried to cut those times, on the same ten contracts and questions.
| Change | Outline, per contract | Per question, picking and answering | Marked clauses quoted |
|---|---|---|---|
| First run | 31 s | 4.3 s | 81% |
| The model names 4 parts instead of up to 8 | 31 s | 4.0 s | 82% |
| The same, on the server’s faster setting | 31 s | 2.6 s | 82% |
| The same, with the labels written eight at a time | 22 s | 2.7 s | 82% |
| A small model (4 billion parameters) writes the labels | 3 s | 3.9 s | 72% |
A question went from 4.3 seconds to 2.7 with the same accuracy, and the outline from 31 seconds per contract to 22. On the faster setting the model drafts two words ahead and checks them, at the price of a shorter maximum document. Writing the labels eight at a time gave less than we expected, because one request already uses most of the graphics card.
The small model wrote the outline ten times faster, but its labels were vaguer, the right part was read less often (88% of questions against 95%), and the result fell back to what the search gave. The outline is worth the larger model’s time.
Limits
- The answering test covers ten contracts, the other test thirty. The differences are large, but the samples are small.
- We did not run the method of the STAIR paper itself, which trains a model. This is a simpler version of its idea.
- The outline was tested inside one contract at a time. Across a whole collection, something must first pick the documents; we did not test that.
- The size of the parts and the length of the labels were our first choice and were not tuned.
- The scoring is ours, as in the first test, and does not compare with published CUAD numbers. The contracts are American and in English.
Questions and answers
- What is an outline of a contract, here?
- The contract is cut into parts of 1,500 characters and the model writes one line per part naming the topics it covers. For each question the model reads those lines and names the parts to read.
- How much better was it than search?
- Reading the same amount of text, the model quoted 81% of the clauses lawyers had marked with the outline, against 71% with the best search we had found. Reading the whole contract gave 82%.
- Does it need training?
- No. The STAIR paper that suggested the idea trains a model; we used the same general model for the labels, the choice of parts and the answer, with no training.
- What does it cost?
- In the first run, 31 seconds of machine time per contract, once, to write the outline, and 4.3 seconds per question. After tuning, 22 seconds per contract and 2.7 seconds per question, with the same accuracy.
- Did any of this use a cloud service?
- No. Everything ran on one workstation with one 16 GB graphics card.
Sources
- Kumar et al., STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation (2026)
- The Atticus Project, CUAD: Contract Understanding Atticus Dataset
- Hendrycks et al., CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review (2021)
- Zhang et al., Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025)
