Promythicus FR root@promythicus.com

Promythicus / Blog

Three ways of reading a contract, joined, beat any one of them

By Bardia Shahrestani, Promythicus Inc. · Published

We had one local AI model read contracts in several different ways. Each way alone quoted 80 to 86% of the clauses lawyers had marked in short contracts, and 60 to 70% in long ones. The ways do not miss the same clauses. Keeping what two or three of them found, with every quote cut from the file by code, gave 87% on short contracts in 2.7 seconds a question and 83% on long ones in 25 seconds, run end to end as one pipeline. The model no longer types its quotes: it names sentence numbers and code pastes the text, so every quote is word for word. A memo drafted from the saved passages carried no invented figure in 30 requests. Nothing was trained.

Summary: three ways of reading joined quote 87% of marked clauses on short contracts and 83% on long ones, against 60 to 84% for each alone; every quote word for word

The problem

A private assistant for a firm has to do three things with the firm’s own documents: answer a question about them, show the passage the answer rests on, and draft from what it found. We measure the first two on CUAD, a public set of 510 commercial contracts in which lawyers marked the clauses that answer 41 standard review questions (is there a cap on liability, can the contract be assigned, which law governs).

Three things were open when this work began:

The test

1. Copy, do not type

The contract is shown to the model with every sentence numbered. The model does not write a quote. It answers with ranges of numbers, “41 to 43”, one for each place, and code pastes the contract’s own text. No quote passes through the model, so none can be altered.

Short contracts, whole contract read for each questionTyping the quotesCopy tool
Marked clauses quoted82.3%85.9%
Quotes word for word90.5%100%
Seconds a question3.31.5

It is more than twice as fast, because the model writes a few numbers instead of hundreds of words, and it is the best single way of reading we have measured. It does say “found” a little more often where the lawyers marked nothing.

2. Readings that differ, joined

We compared, clause by clause, what six methods had quoted on the same contracts. Each scored 80 to 83%. Together they had quoted 89.5% of the clauses: each one misses clauses another finds. So the methods can be joined: keep the passages of two or three, match each to the file’s own words by code, and merge those that overlap.

The three we kept are made in different ways, which is the point:

3. As one pipeline, end to end

Marked clauses quotedShort contractsLong contracts
Copy-tool reading alone83.9%69.6%
Review sheet alone81.5%68.8%
The model’s own search alonenot used60.1%
Reading and sheet87.1%78.7%
All joined87.1%83.0%
Seconds a question2.725

Long contracts by length: 86% from 40,000 to 80,000 characters, 89% from 80,000 to 120,000, and 74% above 120,000. Contracts of fifteen to forty-five pages are now read as well as short ones. The longest remain the weak spot.

Writing the sheet costs about 100 seconds a short contract, once, when it is filed. That is about the time of asking the 41 questions one after another.

The price of joining: more passages

A joined answer returns more. On short contracts, where the lawyers marked nothing for a question, the pipeline still returns a passage more often than one careful reading does, and about a third of all its passages are a marked clause. Some of the rest are fair: the lawyers marked narrowly, and a passage on what happens after termination is not wrong for a question about termination. We did not count how many.

What we do about it: the model is asked, for each passage alone, whether it answers the question, yes or no, and we read how sure it is. Dropping the passages it doubts costs too much (87% falls to 80%). So nothing is dropped. The passages it is sure of are shown as firm, the rest as possibly related. Firm passages are a marked clause two times in three. The label costs 0.4 seconds a question.

4. What it took for long contracts

5. Finding the file first

A question usually names its document loosely: “the supply agreement with Acme”. Among the 510 contracts, 980 such questions:

6. Drafting a memo from what was found

A request such as “draft a short memo on the maintenance agreement with this company: minimum commitment, assignment, expiry, governing law” is done in two steps, the way a person would do it.

Gather and save. The model finds the file and reads it for each point; the passages are saved.

Gathering for a four-point memoMarked clauses savedSeconds a request
Four points in one reading, quotes typed77.7%about 30
One point at a time, quotes typed83.4%27
One point at a time, copy tool84.1%12

Write. At first the model wrote the memo and chose which passages to cite. It cited only some of what it had been given, and one memo of thirty was cut off. So the roles were split: the model writes only the words of each section, and code attaches every saved passage to its section.

The right file was read in 29 requests of 30.

What did not work

Limits

Questions and answers

How can an AI be kept from misquoting a document?
Do not let it type the quote. In our test the contract is shown with numbered sentences, the model answers with ranges of numbers, and code pastes the text. Every quote was word for word, the reading was twice as fast, and it found more clauses than typing did (85.9% against 82.3%).
Why join several readings instead of improving one?
Because they miss different clauses. Six methods each quoted 80 to 83% of the marked clauses on the same contracts; together they had quoted 89.5%. Joining a review sheet written at intake with a reading at the question gave 87% on short contracts, and 83% on long ones with the model’s own search added.
Can a local AI model read a long contract accurately?
Up to about 120,000 characters, yes: 86 to 89% of marked clauses with the joined pipeline, against 52 to 55% for any single method when we began. Above that it reaches 74%.
Does the model invent figures when it drafts a memo?
Not in this test. The model wrote only the wording and code attached the saved passages. Of 61 numbers, dates and amounts in 30 memos, one was not in the cited passages, and it was a section number.

Sources