I Fed 340 Pages of Contracts to GPT-5.6 Sol. A Lawyer Graded Its Work.

The Experiment

I signed a SaaS reseller agreement in June that contained a renewal window I hadn't noticed — 60 days before term end, auto-renewing for 24 months. My lawyer found it in four minutes, billed me for an hour, and I remember thinking: what part of this could an AI have done? So I designed an honest test before drawing conclusions.

I pulled 14 agreements I'd been party to over the last two years — NDAs, vendor MSAs, two employment contracts, a licensing deal, an equipment lease. 340 pages total, real documents, real money. I ran every one through GPT-5.6 Sol with a strict extraction prompt: flag every clause type in a checklist, summarize obligations, identify the five most negotiated terms, and draft questions a lawyer should verify. Then I handed its output to Emma, a commercial attorney at a mid-size firm ($420/hour, for reference), and asked her to mark every finding: correct, overblown, missed, or flat wrong. This is the part where the AI fan club and the AI doomers both get uncomfortable.

What Sol Genuinely Nailed

Clause extraction was boringly excellent. Across 415 clauses on my checklist — indemnification, limitation of liability, IP assignment, termination triggers, non-compete scope, governing law — Sol found 412. The three misses were in a scanned PDF with degraded OCR, which Emma agreed a junior associate might also skim past.

The tabulation was where it shined practically. It built me a comparison table of all 14 agreements: notice periods, cure windows, liability caps as multiples of fees paid, what survives termination. Doing that manually is a junior-associate week. Sol did it in nine minutes for the entire corpus — I timed it — and Emma called the table "document-control accurate" before flagging two characterization issues. It also surfaced subtle cross-references I would have missed: one vendor MSA's SLA credit cap quietly referenced the liability cap in an appendix three signatures away.

Where It Got Dangerous

Three failure classes, in ascending order of how much they'd hurt:

1. Overstated risks. Three of our NDAs allow residuals use of "general knowledge" — industry-standard language. Sol flagged all three as "significant risk to trade secrets." Technically defensible? Emma said honestly she'd raise it. But a business team gets desensitized fast when every page is a fire, and one of the three was boilerplate every competitor uses.
2. Jurisdiction blindness. This is the big one. A German services agreement contained a liability limitation Sol analyzed through a US lens. Under German law, liability for gross negligence can't be capped the way the document implied was possible — so Sol produced a confident, well-formatted paragraph that was quietly wrong in the specific jurisdiction that mattered. It didn't hedge, didn't route itself to counsel, just answered.
3. One flat mischaracterization. An equipment lease clause set early-termination fees at "the greater of $18,000 or remaining payments." Sol summarized it as "capped at remaining payments" — inverted a > into a sentence that read fine. Emma marked it with a red pen and wrote "NO." in the margin. Read the actual numbers on your own contracts, always.

Zero legal citations were hallucinated this time, for what it's worth — I think because it stayed inside the documents and wasn't asked open-world legal questions. Both Emma and I are convinced that's the boundary to respect.

The Lawyer's Scorecard

The minute cost framing that ended our conversation: the same 340 pages through Sol cost me $11.40 in output tokens. Emma's counterpart work would have been 20+ hours. That's not the same product and we both knew it — but it reframes what the expensive hours are for.

The Workflow I Actually Kept

Six weeks later, here's my standing practice. Every contract first goes through Sol with the extraction checklist; I read its clause table myself with a stopwatch, targeting ten minutes. I then hand my lawyer ONLY the flagged items and my ten drafted questions — Emma confirmed this cuts her review scope roughly in half on typical agreements, which is where the real savings live. And one hard rule from both of us: any clause where Sol's confidence feels high but mine feels low gets read character by character before signing. That's the lane where it almost bit me.

If you're evaluating where frontier models are genuinely trustworthy versus benchmark-fabulous, this test slots neatly into the argument I made in my benchmark fine print analysis — headline numbers oversell, structured narrow tasks impress. And if you're considering the data-handling side of feeding contracts into a model API, the compliance section of my enterprise stack guide covers the zero-retention settings you'll want configured first.