Search first or read the whole contract? A local AI tested on 102 contracts
On the standard contract-review test (CUAD), a local model reading the whole contract quoted 82% of the clauses that lawyers had marked. Searching first and reading only the best passages quoted 50% with a basic search and 71% with the best search we found, while reading about a seventh of the text. The search itself went from finding the right clause 62% of the time to 88% with three changes: shorter passages, a newer embedding model, and a reranker. Everything ran on one workstation with one 16 GB graphics card and no cloud.
The question
A private document assistant can answer a question about a contract in two ways. It can read the whole contract every time. Or it can search the contract first and read only the few passages that look relevant. The second way is the only one that works across a whole collection, because no model can read five hundred contracts for one question. We wanted to know what the search costs in accuracy, and how good the search can be made.
The test
- Data: CUAD, a public set of commercial contracts in which lawyers marked the clauses that answer 41 standard review questions (is there a non-compete, a cap on liability, a renewal term, and so on). We used the official test split: 102 contracts and 4,182 questions.
- Machine: one workstation with one 16 GB graphics card. Nothing was sent to a cloud.
- Model: a mixed-precision variant of Qwen 3.8 27B that runs on a single 16 GB GPU.
- Task: for each question, say whether the contract has language on the point, and quote it word for word.
Which search finds the right clause
This part needs no language model. For each of the 1,244 questions that have an answer, across all 102 contracts, we checked whether a passage holding part of the answer was among the ones the search returned. Each search was allowed to return about 6,000 characters.
| Search | Right clause found |
|---|---|
| Basic: an embedding model plus keywords, 1,200-character passages | 62% |
| The same, with the question reduced to the clause name and its description | 69% |
| The same, with 500-character passages | 74% |
| A newer embedding model (Qwen3-Embedding-0.6B), 500-character passages | 83% |
| The same, with a reranker (Qwen3-Reranker-0.6B) re-ordering the first 30 passages | 88% |
Allowed twice as much text, the best search found the right clause 93% of the time.
What did not help: an embedding model seven times larger (the 4B version scored no better than the 0.6B one), adding keyword search to the newer embedding model, putting the contract’s title in front of every passage, and longer passages (2,400 characters dropped to 56%). That shorter passages matter most agrees with what the LegalBench-RAG authors reported. Unlike them, we found that a reranker helps.
What the model then answered
We then let the model answer, on ten contracts short enough to be read whole (410 questions, 248 marked clauses), so that every method faced exactly the same questions.
| Way of reading | Marked clauses quoted | Quotes that are word for word | Text read per question |
|---|---|---|---|
| Basic search, then read | 50% | 99% | about 1,300 tokens |
| Best search, then read | 71% | 98% | about 1,200 tokens |
| Best search, twice as many passages | 73% | 95% | about 2,300 tokens |
| Read the whole contract | 82% | 91% | about 8,400 tokens |
On the simpler question, “does this contract have such a clause at all”, the methods were close: the best search said yes on 93% of the questions that had an answer, the whole-contract reading on 89%. The whole-contract reading raised somewhat fewer false alarms (about a third of its yeses had no marked clause, against about four in ten after a search).
What we take from it
- If one contract fits in the model’s memory, reading all of it is the most thorough. It quoted 82% of the marked clauses, against 71% for the best search.
- The search is where the accuracy is won or lost. The model quoted nearly every marked clause the search put in front of it. Going from the basic search to the best one moved the result from 50% to 71% with the same model and the same amount of reading.
- Shorter reading gives cleaner quotes. After a search, 98% of the quotes matched the contract word for word. Reading the whole contract, 91% did.
- The search is fast. Indexing the ten contracts took 2.4 seconds, and searching one took 0.16 seconds per question. The first question on a whole contract took 8 seconds on average and 16 for the longest; after a search it took 2.5.
So a sensible assistant does both: it searches when the question covers many documents, and reads the whole document when the question is about one that fits.
Limits
- The answering test covers ten contracts of at most 120,000 characters. Ten of the 102 contracts are longer than that and were left out of it. The search test covers all 102.
- The scoring is ours: a quoted passage counts if it overlaps a clause the lawyers marked. It is not the measure used in the CUAD paper, so these numbers cannot be set beside published ones.
- One model, one run. We did not test two newer methods, contextual retrieval and late chunking.
- These are American commercial contracts in English. A firm’s own documents will behave differently, which is why we run the demo on the firm’s own files.
Questions and answers
- Is it better for an AI to read the whole contract or to search it first?
- In our test, reading the whole contract found more: 82% of the clauses lawyers had marked, against 71% for the best search. But reading everything only works for a document that fits in the model’s memory. Across a collection of documents, searching first is the only way.
- What made the search more accurate?
- Three things, in this order: cutting the contracts into shorter passages (500 characters instead of 1,200), using a newer embedding model (Qwen3-Embedding-0.6B), and adding a reranker that re-orders the first 30 passages. Together they took the search from finding the right clause 62% of the time to 88%.
- Does a bigger embedding model search better?
- Not in this test. The 4-billion-parameter version of the same embedding model scored no better than the 0.6-billion one.
- Did any of this use a cloud service?
- No. The model, the search and the contracts were all on one workstation with one 16 GB graphics card.
- Can these numbers be compared with published CUAD results?
- No. We scored by overlap between the model’s quotes and the clauses lawyers marked, which is not the measure used in the CUAD paper.
Sources
- The Atticus Project, CUAD: Contract Understanding Atticus Dataset
- Hendrycks et al., CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review (2021)
- Pipitone and Alami, LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain (2024)
- Zhang et al., Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025)
- Chen et al., M3-Embedding (BGE-M3) (2024)
