Three ways of reading a contract, joined, beat any one of them
We had one local AI model read contracts in several different ways. Each way alone quoted 80 to 86% of the clauses lawyers had marked in short contracts, and 60 to 70% in long ones. The ways do not miss the same clauses. Keeping what two or three of them found, with every quote cut from the file by code, gave 87% on short contracts in 2.7 seconds a question and 83% on long ones in 25 seconds, run end to end as one pipeline. The model no longer types its quotes: it names sentence numbers and code pastes the text, so every quote is word for word. A memo drafted from the saved passages carried no invented figure in 30 requests. Nothing was trained.
The problem
A private assistant for a firm has to do three things with the firm’s own documents: answer a question about them, show the passage the answer rests on, and draft from what it found. We measure the first two on CUAD, a public set of 510 commercial contracts in which lawyers marked the clauses that answer 41 standard review questions (is there a cap on liability, can the contract be assigned, which law governs).
Three things were open when this work began:
- A ceiling. Reading the whole contract for each question quoted 82% of the marked clauses. An outline the model writes itself gave 81%, and the model doing its own search gave 83%. Different methods, the same score.
- Long contracts. Past 40,000 characters (about fifteen pages) every method fell to 52 to 55%.
- Quotes. The model typed its quotes, and 5 to 10% of them were not word for word what the contract says.
The test
- Model: a mixed-precision variant of Qwen 3.8 27B that runs on a single 16 GB GPU. It does every step. Nothing was trained or fitted on contracts.
- Short contracts: ten contracts, 410 questions, 248 marked clauses.
- Long contracts: 117 questions on files of 40,000 to 335,000 characters, 253 marked clauses.
- Drafting: 30 requests for a four-point memo on a contract named loosely among all 510.
- Score: a marked clause counts as quoted when a quote contains it, is contained in it, or shares half its words.
1. Copy, do not type
The contract is shown to the model with every sentence numbered. The model does not write a quote. It answers with ranges of numbers, “41 to 43”, one for each place, and code pastes the contract’s own text. No quote passes through the model, so none can be altered.
| Short contracts, whole contract read for each question | Typing the quotes | Copy tool |
|---|---|---|
| Marked clauses quoted | 82.3% | 85.9% |
| Quotes word for word | 90.5% | 100% |
| Seconds a question | 3.3 | 1.5 |
It is more than twice as fast, because the model writes a few numbers instead of hundreds of words, and it is the best single way of reading we have measured. It does say “found” a little more often where the lawyers marked nothing.
2. Readings that differ, joined
We compared, clause by clause, what six methods had quoted on the same contracts. Each scored 80 to 83%. Together they had quoted 89.5% of the clauses: each one misses clauses another finds. So the methods can be joined: keep the passages of two or three, match each to the file’s own words by code, and merge those that overlap.
The three we kept are made in different ways, which is the point:
- A review sheet written when the contract comes in. The model reads the contract once and notes, for each of the 41 standard points, every passage that deals with it. A later question on one of those points is a lookup: no reading, no graphics card. The idea of having the model write a table and read it back is from Ok and Lee (linked below). Alone: 81.5%, and the best of all at saying whether the contract has such a clause at all.
- A reading at the moment of the question, with the copy tool. Alone: 84 to 86%.
- The model’s own search, for long contracts: it picks twelve passages from its internal states and reads those.
3. As one pipeline, end to end
| Marked clauses quoted | Short contracts | Long contracts |
|---|---|---|
| Copy-tool reading alone | 83.9% | 69.6% |
| Review sheet alone | 81.5% | 68.8% |
| The model’s own search alone | not used | 60.1% |
| Reading and sheet | 87.1% | 78.7% |
| All joined | 87.1% | 83.0% |
| Seconds a question | 2.7 | 25 |
Long contracts by length: 86% from 40,000 to 80,000 characters, 89% from 80,000 to 120,000, and 74% above 120,000. Contracts of fifteen to forty-five pages are now read as well as short ones. The longest remain the weak spot.
Writing the sheet costs about 100 seconds a short contract, once, when it is filed. That is about the time of asking the 41 questions one after another.
The price of joining: more passages
A joined answer returns more. On short contracts, where the lawyers marked nothing for a question, the pipeline still returns a passage more often than one careful reading does, and about a third of all its passages are a marked clause. Some of the rest are fair: the lawyers marked narrowly, and a passage on what happens after termination is not wrong for a question about termination. We did not count how many.
What we do about it: the model is asked, for each passage alone, whether it answers the question, yes or no, and we read how sure it is. Dropping the passages it doubts costs too much (87% falls to 80%). So nothing is dropped. The passages it is sure of are shown as firm, the rest as possibly related. Firm passages are a marked clause two times in three. The label costs 0.4 seconds a question.
4. What it took for long contracts
- The search was not the weak step. For files up to 120,000 characters, 92% of the marked clauses were inside the twelve passages the model read. It quoted 51 to 62% of them. Long contracts have more clauses for each question, and the model stopped after one or two.
- One changed instruction, “the point may be dealt with in several places; quote every one”, took long contracts from 52% to 62%. On short contracts the same instruction does nothing or slightly harms.
- Reading the whole file in parts of about 30,000 characters, each with that instruction, gave 68%, and 70% with the copy tool.
- Joining the reading, the sheet and the search gave 83%.
5. Finding the file first
A question usually names its document loosely: “the supply agreement with Acme”. Among the 510 contracts, 980 such questions:
- A plain word index over each file’s first page, given only the name and the party, puts the right file first 89% of the time and in the first five 98%. The model separates “which file” from “what is asked” in under two seconds.
- Cards the model writes for each file did not beat the file’s own first page.
- Most misses are questions that fit several files, because the same company signed several agreements of the same kind. With one candidate the right file is picked 93% of the time, with several 61%. The software should show the candidates and ask, not guess.
6. Drafting a memo from what was found
A request such as “draft a short memo on the maintenance agreement with this company: minimum commitment, assignment, expiry, governing law” is done in two steps, the way a person would do it.
Gather and save. The model finds the file and reads it for each point; the passages are saved.
| Gathering for a four-point memo | Marked clauses saved | Seconds a request |
|---|---|---|
| Four points in one reading, quotes typed | 77.7% | about 30 |
| One point at a time, quotes typed | 83.4% | 27 |
| One point at a time, copy tool | 84.1% | 12 |
Write. At first the model wrote the memo and chose which passages to cite. It cited only some of what it had been given, and one memo of thirty was cut off. So the roles were split: the model writes only the words of each section, and code attaches every saved passage to its section.
- 30 memos of 30 written, in 8.7 seconds each.
- Every cited passage is word for word in the contract.
- Of 61 numbers, dates and amounts in the memos’ own wording, one was not in the cited passages: a section number. No figure about a contract was invented.
- All 19 sections with no saved passage said the contract was not found to deal with the point.
The right file was read in 29 requests of 30.
What did not work
- Notes written with no question in mind. The model wrote a line on each passage naming every legal point in it, then chose passages from those lines: 79.8%, the lowest of all.
- The written table as a file finder: 76 to 80% right file first, against 88% for the plain word index.
- Writing the sheet with the copy tool too. Long contracts fell from 83% to 75%. Two readings made the same way miss the same clauses. The join gains from methods that differ.
- Reading whole or in parts. We expected the copy tool to do better reading a 100,000-character contract in one call than in parts. It made no difference: 74.4% against 73.9%, and identical once joined.
Limits
- One run of each test. With about 250 marked clauses, two points either way is noise.
- English commercial contracts and CUAD’s 41 review points. The sheet answers only the points it was written for, and the test takes the question’s point as known. The other two readings take any question.
- On long contracts the search step uses the passages from its own earlier run; its time is counted.
- The memo check covers what is cited and the figures. A sentence can cite a real passage and still misstate it; that is not measured.
- Our scoring. These numbers cannot be set beside published CUAD results.
Questions and answers
- How can an AI be kept from misquoting a document?
- Do not let it type the quote. In our test the contract is shown with numbered sentences, the model answers with ranges of numbers, and code pastes the text. Every quote was word for word, the reading was twice as fast, and it found more clauses than typing did (85.9% against 82.3%).
- Why join several readings instead of improving one?
- Because they miss different clauses. Six methods each quoted 80 to 83% of the marked clauses on the same contracts; together they had quoted 89.5%. Joining a review sheet written at intake with a reading at the question gave 87% on short contracts, and 83% on long ones with the model’s own search added.
- Can a local AI model read a long contract accurately?
- Up to about 120,000 characters, yes: 86 to 89% of marked clauses with the joined pipeline, against 52 to 55% for any single method when we began. Above that it reaches 74%.
- Does the model invent figures when it drafts a memo?
- Not in this test. The model wrote only the wording and code attached the saved passages. Of 61 numbers, dates and amounts in 30 memos, one was not in the cited passages, and it was a section number.
