Contract Review at Scale: Building Legal Document Intelligence
Short answer
Reliable AI contract review needs more than chunk-and-embed RAG. We parse the document's structure, map how clauses reference each other, compare each clause against a configurable playbook, and grade deviations with a written rationale. Every decision is logged with source text and confidence so lawyers can verify it.
Key takeaways
- Naive chunking breaks legal meaning because clauses modify each other across sections.
- Hierarchical parsing plus a clause-relationship graph improves precision.
- An append-only audit trail with source text and rationale is what makes legal AI defensible.
Legal document review is one of those tasks where AI confidence matters as much as AI capability. Get a clause wrong, and a client might sign away rights they meant to retain.
That constraint shaped every technical decision we made when building our Legal Document Intelligence system.
The Problem With Naive RAG
The first instinct most people have is to chunk the document, embed it, and run similarity search against a playbook. We tried this. The recall was acceptable. The precision wasn't.
The issue: legal meaning is highly context-dependent. A termination clause buried in section 14.3 might directly modify a payment term in section 4.1. Naive chunking severs those relationships.
What We Built
We use a hierarchical parsing approach. First, we extract the document's structural outline using Unstructured and LlamaParse. Then we build a semantic graph of clause relationships before we do any extraction.
The risk-flagging step compares extracted clauses against a configurable playbook stored in a vector database. But it's not just similarity — we run a secondary classification pass that rates deviation severity (low / medium / high) with a mandatory rationale.
Every AI decision is logged to an append-only audit table. Every clause extraction includes the source text, the confidence score, and the model's reasoning. That audit trail isn't optional — it's what makes the system legally defensible.
Results
On a lengthy investment agreement, the system surfaced a focused set of clauses for attention in minutes — and the client's legal team validated them as genuine issues worth a closer look. Across our test contracts, the pipeline held up where naive RAG fell down.
That's not magic. That's careful prompt engineering layered on top of a well-designed data pipeline.
Frequently asked questions
Can AI review contracts accurately?
AI can reliably extract clauses and flag deviations from a playbook when it is built on structured parsing, clause-relationship mapping and human review. It should support lawyers, not replace their judgment.
Why doesn't standard RAG work well for contracts?
Standard RAG splits documents into chunks, which separates clauses that depend on each other, such as a termination clause that changes a payment term. Retrieval then misses the context that determines the clause's meaning.
What is a contract playbook in AI review?
A playbook is your organization's standard positions and acceptable fallbacks for each clause type. The AI compares each extracted clause to it and flags how far the contract deviates.
How do you make legal AI auditable?
Log every extraction and flag with the source text, model reasoning, confidence score and model version in an append-only store, so any output can be traced and checked.